HMI display test method and device, electronic equipment and storage medium

Abstract

The invention provides an HMI display test method and device, electronic equipment and a storage medium, and the HMI display test method comprises the steps: starting an automatic test, and controlling HMI equipment to display preset elements; obtaining a to-be-processed image obtained by shooting the HMI equipment by the camera; acquiring identification data of the to-be-processed image; wherein the identification data is obtained by utilizing a pre-trained element identification model according to the to-be-processed image, and the identification data comprises information of a target element in the to-be-processed image; and performing comparison verification according to the information of the target element and the information of the preset element to obtain a display test result of the HMI equipment. According to the scheme, the HMI equipment is automatically controlled to display the preset element, the image displayed by the HMI equipment is captured through the camera, and the target element in the image is intelligently recognized through the pre-trained element recognition model, so that the whole-process automatic test of HMI equipment display is completed, and the efficiency and accuracy of HMI display test are improved.

Description

Technical Field

[0001] The present application relates to the field of industrial automation technology, and in particular to a display test method, device, electronic device and storage medium for an HMI. Background Art

[0002] Human Machine Interface (HMI) products are human-machine interaction devices that include hardware and software, which realize information exchange between people and machines, such as text or graphic display and input functions. Usually, the hardware part includes processors, displays, memory, input units, communication interfaces, etc., and the software part includes system software and application software running in HMI hardware, such as screen configuration software. HMI products are widely used in the industrial field because they provide rich device information and real-time data display.

[0003] During the development and production of HMI products, the display of HMI products needs to be tested to ensure the display accuracy. At present, HMI display testing is done manually, and testers manually verify the content displayed by HMI products by looking at them, which results in low efficiency of display testing. Summary of the invention

[0004] In view of this, the HMI display test method, device, electronic device and storage medium provided in the present application can complete the full-process automated test of the HMI device display, avoid manual participation in the test, and improve the efficiency and accuracy of the HMI display test.

[0005] According to a first aspect of an embodiment of the present application, a display test method for an HMI is provided, which is applied to a first electronic device, and the method includes: starting an automated test to control the HMI device to display preset elements; obtaining an image to be processed obtained by photographing the HMI device with a camera; obtaining recognition data of the image to be processed; wherein the recognition data is obtained based on the image to be processed using a pre-trained element recognition model, and the recognition data includes information of a target element in the image to be processed; and comparing and verifying the information of the target element and the information of the preset element to obtain a display test result of the HMI device.

[0006] According to a second aspect of an embodiment of the present application, a display test method for an HMI is provided, which is applied to a second electronic device, and the method includes: receiving an image to be processed sent by a first electronic device; the image to be processed is an image obtained by photographing the HMI device by a camera when the first electronic device controls the HMI device to display preset elements; obtaining recognition data of the image to be processed based on the image to be processed using a pre-trained element recognition model; the recognition data includes information of a target element in the image to be processed; and sending the recognition data to the first electronic device.

[0007] According to a third aspect of an embodiment of the present application, a display test device for an HMI is provided, comprising: a control module, used to start an automated test, and control the HMI device to display preset elements; a first acquisition module, used to acquire an image to be processed obtained by photographing the HMI device with a camera; a second acquisition module, used to acquire recognition data of the image to be processed; wherein the recognition data is obtained based on the image to be processed using a pre-trained element recognition model, and the recognition data includes information of a target element in the image to be processed; and a verification module, used to compare and verify the information of the target element with the information of the preset element, and obtain a display test result of the HMI device.

[0008] According to a fourth aspect of an embodiment of the present application, a display test device for an HMI is provided, comprising: a receiving module, used to receive an image to be processed sent by a first electronic device; the image to be processed is an image obtained by photographing the HMI device by a camera when the first electronic device controls the HMI device to display preset elements; a recognition module, used to obtain recognition data of the image to be processed based on the image to be processed using a pre-trained element recognition model; the recognition data includes information of a target element in the image to be processed; and a sending module, used to send the recognition data to the first electronic device.

[0009] According to the fifth aspect of the embodiments of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a bus, wherein the processor, the memory and the communication interface communicate with each other through the bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method described in the first aspect or the second aspect above.

[0010] According to the sixth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes the method described in the first aspect or the second aspect.

[0011] According to the seventh aspect of the embodiments of the present application, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes computer-executable instructions, which, when executed, enable at least one processor to perform the method described in the first or second aspect above.

[0012] It can be seen from the above technical solution that the first electronic device can communicate with the camera, the HMI device, and the second electronic device. The first electronic device starts the automated test and controls the HMI device to display the preset elements. The first electronic device obtains the image to be processed including the display interface of the HMI device taken by the camera and sends it to the second electronic device. The second electronic device uses the pre-trained element recognition model to process the image to be processed to obtain recognition data, and sends the recognition data to the first electronic device, so that the first electronic device compares and verifies the recognition data and the information of the preset elements to obtain whether the display test result of the HMI device is correct. By automatically controlling the HMI device to display the preset elements, capturing the image displayed by the HMI device through the camera, and intelligently identifying the target elements in the image through the pre-trained element recognition model, the full automated test of the HMI device display is completed, avoiding manual participation in the test process, and improving the efficiency and accuracy of the HMI display test. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1A is a schematic diagram of a display test system provided in an embodiment of the present application;

[0014] Figure 1B is another schematic diagram of a display test system provided in an embodiment of the present application;

[0015] Figure 2 It is a flow chart of the HMI display test method provided in the embodiment of the present application;

[0016] Figure 3 It is a schematic diagram of preset elements displayed by the HMI device provided in the embodiment of the present application;

[0017] Figure 4 is another flow chart of the HMI display test method provided in the embodiment of the present application;

[0018] Figure 5 It is a schematic diagram of a display test device of an HMI provided in an embodiment of the present application;

[0019] Figure 6 is another schematic diagram of the HMI display test device provided in an embodiment of the present application;

[0020] Figure 7 It is a schematic diagram of an electronic device provided in an embodiment of the present application.

[0021] The reference numerals are as follows:

[0022] 201: Start the automated test and control the HMI device to display the preset elements

[0023] 202: Obtain the image to be processed obtained by shooting the HMI device with the camera

[0024] 203: Get the recognition data of the image to be processed

[0025] 204: Compare and verify the target element information with the preset element information to obtain and display the test results

[0026] 2031: Obtain recognition data of the image to be processed using the pre-trained element recognition model according to the image to be processed

[0027] 2036: Sending the image to be processed to the second electronic device

[0028] 2037: receiving identification data sent by the second electronic device

[0029] 401: Obtain recognition data of the image to be processed using the pre-trained element recognition model according to the image to be processed

[0030] 100: display test system 101: first electronic device 102: second electronic device

[0031] 103: HMI device 104: Camera 200: HMI display test method

[0032] 301: Display frame of HMI device 302: Reference frame 400: Display test method of HMI

[0033] 500: HMI display test device 501: control module 502: first acquisition module

[0034] 503: Second acquisition module 504: Verification module 600: HMI display test device

[0035] 601: Receiving module 602: Identifying module 603: Sending module

[0036] 700: electronic device 702: processor 704: communication interface

[0037] 706: memory 708: bus 710: program DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be described below in conjunction with the accompanying drawings.

[0039] As mentioned above, in order to ensure the display accuracy of HMI products, it is necessary to perform display testing on HMI products. Currently, the test is done manually, where testers manually verify the content displayed on the HMI products by looking at them, which results in low efficiency of display testing.

[0040] In the embodiment of the present application, the display test system includes an HMI device, a camera and an electronic device. The electronic device can communicate with the camera and the HMI device to transmit relevant data. The automated test is started on the electronic device to control the HMI device to display the preset element. The camera shoots the HMI device to obtain an image to be processed, and the image to be processed includes the element actually displayed by the HMI device, which is called the target element. The camera sends the image to be processed to the electronic device, and the electronic device processes the image to be processed using a pre-trained element recognition model to obtain recognition data, and the recognition data includes information of the target element in the image to be processed. The electronic device compares and verifies the recognition data and the information of the preset element, thereby obtaining whether the display test result of the HMI device is correct or not. It can be seen that by building an automated test environment, the HMI device can be controlled to display the preset element, and the image including the display interface of the HMI device can be obtained by shooting with the camera, and the elements in the image can be intelligently identified by the pre-trained element recognition model to obtain the element recognition result, thereby, according to the information of the preset element expected to be displayed and the information of the target element actually displayed, the full automated test of the HMI device display is completed, and no human participation in the test process is required, which improves the efficiency and accuracy of the HMI display test.

[0041] Next, through Figure 1A and Figure 1B A display test system is exemplarily described.

[0042] In one implementation, Figure 1A Schematic diagram of a display test system provided in an embodiment of the present application. Figure 1AAs shown, the display test system 100 includes a first electronic device 101, a second electronic device 102, an HMI device 103 and a camera 104. The first electronic device 101 can communicate with the second electronic device 102, the HMI device 103 and the camera 104. The camera 104 is set at a fixed position according to the position of the HMI device 103, and the HMI device 103 is located within the field of view of the camera 104, so that the image captured by the camera 104 includes the interface actually displayed by the HMI device 103. The first electronic device 101 can run an automated test script to control the HMI device 103 to display preset elements, control the camera 104 to capture the image to be processed containing the display interface of the HMI device 103, and complete the subsequent comparison and verification based on the information of the target element and the information of the preset element by running the automated test script to obtain the display test result of the HMI device. A pre-trained element recognition model is deployed on the second electronic device 102. The first electronic device 101 sends the image to be processed captured by the camera 104 to the second electronic device 102 , so that the second electronic device 102 processes the image to be processed using the element recognition model to obtain recognition data, and sends the recognition data to the first electronic device 101 for comparison and verification.

[0043] By setting up a first electronic device 101 and a second electronic device 102, the first electronic device 101 is connected to communicate with the HMI device 103 and the camera 104, and an automated test script is run, and the second electronic device 102 deploys an element recognition model, different functions are deployed on different electronic devices in the display test system, and there is no need to deploy the same complex test environment on each electronic device, which simplifies the complexity of setting up the test system and makes the test environment adjustment more flexible.

[0044] It should be noted that the present application does not limit the number of the first electronic device 101, the second electronic device 102, the HMI device 103 and the camera 104. For example, there are multiple HMI devices 103, and the HMI devices 103 are provided with cameras 104 accordingly.

[0045] In another implementation, Figure 1B is another schematic diagram of the display test system provided in the embodiment of the present application. Figure 1B As shown, the display testing system 100 includes a first electronic device 101 , an HMI device 103 , and a camera 104 . Figure 1B It can be understood as: Figure 1A The first electronic device 101 and the second electronic device 102 in the embodiment are the same device. Figure 1B In the embodiment, the first electronic device 101 can run an automated test script and deploy a pre-trained element recognition model. The display test system provided by this implementation involves a relatively small number of electronic devices.

[0046] The implementation methods of the present application are described in detail below with reference to the accompanying drawings.

[0047] Figure 2 1 is a flow chart of the HMI display test method provided in the embodiment of the present application. The execution subject of this embodiment involves the first electronic device 101 and may also involve the second electronic device 102. Figure 2 As shown, the HMI display test method 200 provided in this embodiment includes:

[0048] Step 201: The first electronic device 101 starts an automated test and controls an HMI device to display preset elements.

[0049] The first electronic device 101 runs the automated test script, starts the automated test, communicates with the HMI device and the camera, and controls the HMI device to display preset elements. This embodiment does not limit the communication protocol for the first electronic device 101 to communicate with the HMI device and the camera, and can use a wired connection or a wireless connection.

[0050] For the sake of distinction, the elements expected to be displayed on the HMI device when the automation script is run are referred to as preset elements. Optionally, the elements include text and / or images (also called icons).

[0051] For example, Figure 3 Schematic diagram of preset elements displayed by the HMI device provided in the embodiment of the present application. Figure 3 As shown, the display interface of the HMI device includes 3 rows. The preset elements in the first row include, from left to right: the image "temperature", the text "36.6℃", the image "settings", and the image "house". The preset elements in the second row include, from left to right: the image "left button", the image "fan", and the image "right button". The preset elements in the third row include, from left to right: the image "left button", the image "time", and the image "right button".

[0052] Step 202: The first electronic device 101 obtains an image to be processed obtained by photographing the HMI device with a camera.

[0053] The image to be processed includes the elements actually displayed by the HMI device. For the sake of distinction, the elements actually displayed by the HMI device are referred to as target elements. It is understood that the content and display position of the preset element and the target element may be the same or different, so verification is required to determine whether the display of the HMI device is correct.

[0054] Step 203: The first electronic device 101 obtains recognition data of the image to be processed, wherein the recognition data is obtained based on the image to be processed using a pre-trained element recognition model, and the recognition data includes information of a target element in the image to be processed.

[0055] Optionally, the element recognition model includes a text recognition model and an image recognition model, the text recognition model is used to recognize text in the image to be processed, and the image recognition model is used to recognize images in the image to be processed.

[0056] In this implementation, if the type of target element in the image to be processed only includes text type, the text recognition model is used to identify the target element of text type in the image to be processed, and the text recognition result is obtained as the recognition data of the image to be processed. If the type of target element in the image to be processed only includes image type, the image recognition model is used to identify the target element of image type in the image to be processed, and the image recognition result is obtained as the recognition data of the image to be processed. If the type of target element in the image to be processed includes text type and image type, the text recognition model is used to identify the target element of text type in the image to be processed to obtain the text recognition result, and the image recognition model is used to identify the target element of image type in the image to be processed to obtain the image recognition result, and the text recognition result and the image recognition result are used as the recognition data of the image to be processed.

[0057] By using a text recognition model and an image recognition model to perform text recognition and image recognition respectively, the accuracy of element recognition is improved.

[0058] The target element information includes the value and position of the target element. If the target element is text, the value of the target element includes but is not limited to text, numbers or characters. For example, Figure 3 , recognize that the value of the text element in the first row is "36.6". If the target element is an image, the value of the target element includes an image ID or an image name, which can uniquely distinguish different images. Figure 3 Assume that the image identifier of the image "temperature" is 101, and the image name is "temp"; the image identifier of the image "setting" is 102, and the image name is "setting"; the image identifier of the image "house" is 103, and the image name is "InDoor". The value of the identified image element "temperature" is "temp". The value of the target element can be used to determine whether the content displayed by the element is correct. Among them, the image to be processed includes a reference frame, and the position of the target element is the position determined according to the reference frame. The position of the target element can be used to determine whether the display position of the element in the interface of the HMI device is correct. Optionally, the information of the target element also includes confidence. The higher the confidence, the more credible the information of the target element is.

[0059] Optionally, the target element's information is in JavaScript Object Notation (JSON) data format. JSON is a lightweight data exchange format that can convert a set of data into a string, so that the string can be easily transferred on the network, between devices, or between programs, and when needed, it can be restored to a data format supported by various programming languages ​​for easy data processing.

[0060] Optionally, in one implementation, in step 203, the first electronic device 101 obtains identification data of the image to be processed, including:

[0061] Step 2036 : The first electronic device 101 sends the image to be processed to the second electronic device 102 .

[0062] Step 2037 : The first electronic device 101 receives the identification data sent by the second electronic device 102 .

[0063] In this implementation, the first electronic device 101 is not deployed with an element recognition model, and the second electronic device 102 is deployed with an element recognition model. The first electronic device 101 and the second electronic device 102 are different electronic devices. Figure 1A The structure of the display test system is shown.

[0064] Optionally, in step 2036, the first electronic device 101 sending the to-be-processed image to the second electronic device 102 may include:

[0065] The first electronic device 101 sends a POST request to the second electronic device 102 according to Hypertext Transfer Protocol (HTTP), where the POST request includes an image to be processed.

[0066] The HTTP protocol can be used to transmit data between devices. It is an application layer protocol that communicates based on the TCP / IP protocol family. In this embodiment, the HTTP protocol allows a first electronic device to send a request to a second electronic device and receive a response from the second electronic device, thereby achieving information interaction. Among them, a POST request is an HTTP request method that transfers data through a request body.

[0067] Optionally, in another implementation, in step 203, the first electronic device 101 obtains identification data of the image to be processed, including:

[0068] Step 2031: The first electronic device 101 obtains recognition data of the image to be processed using a pre-trained element recognition model according to the image to be processed.

[0069] In this implementation, an element recognition model is deployed on the first electronic device 101, which can be seen in Figure 1B The structure of the display test system is shown.

[0070] Step 204: The first electronic device 101 compares and verifies the information of the target element and the information of the preset element to obtain a display test result of the HMI device.

[0071] The first electronic device 101 continues to run the automated test script, performs comparison and verification based on the information of the target element and the information of the preset element, and determines whether the content and position of the element display are correct, thereby obtaining a display test result of the HMI device.

[0072] In one implementation, the information of the preset element includes the value and position of the preset element, and the information of the target element includes the value and position of the target element. In step 204, the first electronic device 101 compares and verifies the information of the target element and the information of the preset element to obtain a display test result of the HMI device, including:

[0073] The value of the target element is compared with the value of the preset element to obtain a value comparison result, and the position of the target element is compared with the position of the preset element to obtain a position comparison result.

[0074] The display test result of the HMI device is determined according to the value comparison result and the position comparison result.

[0075] Optionally, the value comparison result and the position comparison result may have a small error tolerance. For example, if the difference between the position of the target element and the position of the preset element is within a preset position error, it is determined that the display positions of the target element and the preset element are the same.

[0076] In another implementation, the information of the target element further includes the confidence level of the target element. In step 204, the first electronic device 101 compares and verifies the information of the target element with the information of the preset element to obtain a display test result of the HMI device, including:

[0077] If the confidence of the target element is greater than a preset threshold, the value of the target element is compared with the value of the preset element to obtain a value comparison result, and the position of the target element is compared with the position of the preset element to obtain a position comparison result;

[0078] The display test result of the HMI device is determined according to the value comparison result and the position comparison result.

[0079] In this implementation, the information of the target element whose confidence level is greater than a preset threshold is used for comparison and verification, thereby further improving the accuracy of the display test.

[0080] It should be noted that during the automated test process, the HMI device can display one or more interfaces, and the preset elements in different interfaces can be different. For any interface, the camera can shoot the HMI device to obtain one or more images to be processed, and the automated test results of the HMI display are obtained by executing the HMI display test method provided in this embodiment.

[0081] It can be seen that the test method for HMI display provided in this embodiment, the display test system built includes an HMI device, a camera and a first electronic device, and may also include a second electronic device. The first electronic device can communicate with the camera, the HMI device and the second electronic device to transmit relevant data. The first electronic device starts the automated test and controls the HMI device to display the preset element. The camera shoots the HMI device to obtain the image to be processed, and the image to be processed includes the target element actually displayed by the HMI device. The camera sends the image to be processed to the first electronic device, and the first electronic device obtains the recognition data of the image to be processed. The recognition data is obtained according to the image to be processed using the pre-trained element recognition model, so that the first electronic device compares and verifies the recognition data and the information of the preset element, so as to obtain whether the display test result of the HMI device is correct or not. By building an automated test environment, controlling the HMI device to display the preset element, obtaining the image to be processed including the display interface of the HMI device by shooting with the camera, and obtaining the element recognition result and verifying it by the pre-trained element recognition model, the full automated test of the HMI device display is completed, and no manual participation in the test process is required, thereby improving the efficiency and accuracy of the HMI display test.

[0082] Another embodiment of the present application further provides a display test method for an HMI, which illustrates the first electronic device 101 acquiring identification data of an image to be processed in step 203. Figure 1A The display test system shown is described as an example, and the execution subject involves the first electronic device 101 and the second electronic device 102 . Figure 4 400 is another flow chart of the HMI display test method provided in the embodiment of the present application. The HMI display test method 400 provided in this embodiment may include:

[0083] Step 2036 : The second electronic device 102 receives the image to be processed sent by the first electronic device 101 .

[0084] The image to be processed is an image obtained by shooting the HMI device with a camera when the first electronic device 101 controls the HMI device to display preset elements. Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0085] Optionally, a POST request sent by the first electronic device 101 is received according to the HTTP protocol, where the POST request includes the image to be processed.

[0086] Step 401: The second electronic device 102 obtains recognition data of the image to be processed using a pre-trained element recognition model according to the image to be processed, wherein the recognition data includes information of a target element in the image to be processed.

[0087] Among them, the element recognition model, target element and information of target element can be found in Figure 2 The relevant description of the illustrated embodiment will not be repeated here.

[0088] Step 2037 : The second electronic device 102 sends the identification data to the first electronic device 101 .

[0089] The test method for HMI display provided in this embodiment is a display test system that includes an HMI device, a camera, a first electronic device, and a second electronic device. The first electronic device can communicate with the camera, the HMI device, and the second electronic device to transmit relevant data. The first electronic device starts an automated test to control the HMI device to display preset elements. The camera shoots the HMI device to obtain an image to be processed, and the image to be processed includes the target element actually displayed by the HMI device. The camera sends the image to be processed to the first electronic device, and the first electronic device sends the image to be processed to the second electronic device. The second electronic device uses a pre-trained element recognition model to process the image to be processed to obtain recognition data, and sends the recognition data to the first electronic device, so that the first electronic device compares and verifies the recognition data and the information of the preset element to obtain whether the display test result of the HMI device is correct. By building an automated testing environment, the first electronic device runs an automated testing script, controls the HMI device to display preset elements, obtains a to-be-processed image including the HMI device display interface through camera photography, and the second electronic device obtains an element recognition result through a pre-trained element recognition model. The first electronic device continues to run the automated testing script to verify the recognition data, thereby completing the full automated testing of the HMI device display without the need for human participation in the testing process, thereby improving the efficiency and accuracy of the HMI display test.

[0090] Optionally, the target element information is in json data format.

[0091] Optionally, in step 401, the second electronic device 102 obtains recognition data of the image to be processed using a pre-trained element recognition model according to the image to be processed, including:

[0092] Get the position of the reference bounding box in the image to be processed.

[0093] The image to be processed and the position of the reference frame are input into the element recognition model, and the text recognition result and / or image recognition result are output. The text recognition result includes the value, position and confidence of the target element of the text type in the image to be processed, and the image recognition result includes the value, position and confidence of the target element of the image type in the image to be processed.

[0094] Generate recognition data in JSON format based on text recognition results and / or image recognition results.

[0095] In this implementation, the reference frame is used to locate the position of the target element in the image to be processed. Optionally, the reference frame can be a frame actually displayed in the display interface of the HMI device. For example, Figure 3 As shown, 301 is the display frame of the HMI device, and 302 is the reference frame in the display interface of the HMI device. The position of the reference frame in the image to be processed can be identified by an image processing method, and this embodiment does not limit the identification method. Optionally, in one implementation, if the reference frame actually displayed in the display interface of the HMI device is not identified, the default frame is determined as the reference frame. For example, when setting up a test environment, the relative position of the HMI device and the camera is obtained, and the default frame is determined according to the relative position of the HMI device and the camera. Optionally, since the position between the HMI device and the camera is relatively fixed, for multiple images to be processed captured by the camera, the position of the reference frame in any image to be processed can be obtained without determining the position of the reference frame in each image to be processed. Afterwards, the position of the image to be processed and the reference frame is input into the element recognition model, and the text recognition result and / or the image recognition result are output, and the recognition data in the json data format is generated according to the text recognition result and / or the image recognition result. Optionally, if the element recognition model outputs text recognition results and image recognition results, the text recognition results and image recognition results can be integrated into a file in a json data format; or, the text recognition results and image recognition results can be integrated into different files in different json data formats.

[0096] Optionally, the element recognition model includes a text recognition model and an image recognition model. The position of the image to be processed and the reference frame are input into the element recognition model, and the text recognition result and / or the image recognition result are output, including:

[0097] Determine the type of target element in the image to be processed;

[0098] If the type of the target element includes a text type, the image to be processed and the position of the reference frame are input into the text recognition model, and the text recognition result is output;

[0099] If the type of the target element includes an image type, the position of the image to be processed and the reference frame are input into the image recognition model, and the image recognition result is output.

[0100] In this implementation, a text recognition model and an image recognition model are used to perform text recognition and image recognition respectively according to the type of target elements in the image to be processed, thereby improving the accuracy of element recognition.

[0101] Corresponding to the above method embodiment, Figure 5 Schematic diagram of a display test device for an HMI provided in an embodiment of the present application. Figure 5 As shown, the display test device 500 of the HMI includes:

[0102] The control module 501 is used to start the automated test and control the HMI device to display preset elements;

[0103] A first acquisition module 502 is used to acquire an image to be processed obtained by photographing the HMI device with a camera;

[0104] The second acquisition module 503 is used to acquire recognition data of the image to be processed; wherein the recognition data is obtained according to the image to be processed using a pre-trained element recognition model, and the recognition data includes information of a target element in the image to be processed;

[0105] The verification module 504 is used to compare and verify the information of the target element and the information of the preset element to obtain a display test result of the HMI device.

[0106] The HMI display test device 500 provided in the embodiment of the present application can be used to execute the steps executed by the first electronic device 101 in the aforementioned method embodiment. Since the information interaction, execution process, etc. between the various units in the HMI display test device 500 are based on the same concept as the aforementioned HMI display test method embodiment, the specific contents can be found in the description in the aforementioned HMI display test method embodiment, and will not be repeated here.

[0107] Corresponding to the above method embodiment, Figure 6 is another schematic diagram of the HMI display test device provided in the embodiment of the present application. Figure 6 As shown, the display test device 600 of the HMI includes:

[0108] The receiving module 601 is used to receive an image to be processed sent by a first electronic device; the image to be processed is an image obtained by shooting the HMI device with a camera when the first electronic device controls the HMI device to display preset elements;

[0109] The recognition module 602 is used to obtain recognition data of the image to be processed by using a pre-trained element recognition model according to the image to be processed; the recognition data includes information of a target element in the image to be processed;

[0110] The sending module 603 is used to send the identification data to the first electronic device.

[0111] The HMI display test device 600 provided in the embodiment of the present application can be used to execute the steps executed by the second electronic device 102 in the aforementioned method embodiment. Since the information interaction, execution process, etc. between the various units in the HMI display test device 600 are based on the same concept as the aforementioned HMI display test method embodiment, the specific contents can be found in the description in the aforementioned HMI display test method embodiment, and will not be repeated here.

[0112] Figure 7 1 is a schematic diagram of an electronic device provided in an embodiment of the present application. The electronic device may be the first electronic device 101 or the second electronic device 102 in the above method embodiment. The specific embodiment of the present application does not limit the specific implementation of the electronic device. Figure 7 The electronic device 700 provided in the embodiment of the present application includes: a processor 702, a communication interface 704, a memory 706, and a bus 708. Among them:

[0113] The processor 702 , the communication interface 704 , and the memory 706 communicate with each other via a bus 708 .

[0114] The communication interface 704 is used to communicate with other electronic devices or servers.

[0115] The processor 702 is used to execute the program 710, and specifically can execute the relevant steps in the above-mentioned HMI display test method embodiment.

[0116] Specifically, the program 710 may include program codes, which include computer operation instructions.

[0117] The processor 702 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0118] The memory 706 is used to store the program 710. The memory 706 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0119] The program 710 may be specifically used to enable the processor 702 to execute the HMI display test method in any of the aforementioned embodiments.

[0120] The specific implementation of each step in program 710 can refer to the corresponding description of the corresponding steps and units in the above-mentioned HMI display test method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0121] The present application also provides a computer-readable storage medium storing instructions for causing a machine to execute the display test method of the HMI as described herein. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0122] In this case, the program code read from the storage medium itself can implement the function of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of the present application.

[0123] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0124] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0125] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0126] The embodiment of the present application also provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer executable instructions, which, when executed, cause at least one processor to execute the HMI display test method provided in the above embodiments. It should be understood that each solution in this embodiment has the corresponding technical effect in the above method embodiment, which will not be repeated here.

[0127] The present application also provides a display test system, including an HMI device, a camera, and a first electronic device. Optionally, a second electronic device may also be included. The first electronic device and the second electronic device are used to perform the HMI display test method provided in the above embodiments, which can be seen in Figure 1A or Figure 1B , I will not go into details here.

[0128] It should be noted that not all steps and modules in the above-mentioned processes and device structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.

[0129] Nouns and pronouns relating to persons in this patent application are not limited to a specific gender.

[0130] In the above embodiments, the hardware module can be implemented mechanically or electrically. For example, a hardware module can include permanent dedicated circuits or logic (such as special processors, FPGA or ASIC) to complete the corresponding operation. The hardware module can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete the corresponding operation. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0131] The present application is presented and described in detail above through the accompanying drawings and preferred embodiments; however, the present application is not limited to these disclosed embodiments, and those skilled in the art may know based on the above-mentioned multiple embodiments that the code review methods in the above-mentioned different embodiments may be combined to obtain more embodiments of the present application, and these embodiments are also within the protection scope of the present application.

Claims

1. A display test method (200) for an HMI, characterized in that: Applied to a first electronic device (101), the method comprises: Starting the automated test and controlling the human-machine interface HMI device (103) to display preset elements; Acquiring an image to be processed obtained by photographing the HMI device with a camera (104); Acquire recognition data of the image to be processed; wherein the recognition data is obtained according to the image to be processed using a pre-trained element recognition model, and the recognition data includes information of a target element in the image to be processed; A comparison and verification is performed based on the information of the target element and the information of the preset element to obtain a display test result of the HMI device.

2. The method according to claim 1, characterized in that The target element information is in json data format, and the target element information includes the value, position and confidence of the target element.

3. The method according to claim 2, characterized in that The image to be processed includes a reference frame, and the position of the target element is a position determined according to the reference frame.

4. The method according to claim 2 or 3, characterized in that: The information of the preset element includes the value and position of the preset element; and the comparing and verifying the information of the target element and the information of the preset element to obtain the display test result of the HMI device includes: If the confidence of the target element is greater than a preset threshold, the value of the target element is compared with the value of the preset element to obtain a value comparison result, and the position of the target element is compared with the position of the preset element to obtain a position comparison result; A display test result of the HMI device is determined according to the value comparison result and the position comparison result.

5. The method according to any one of claims 1 to 3, characterized in that: The obtaining of identification data of the image to be processed includes: Sending the image to be processed to a second electronic device (102); The identification data sent by the second electronic device is received.

6. The method according to claim 5, characterized in that The sending the to-be-processed image to the second electronic device comprises: A POST request is sent to the second electronic device according to the Hypertext Transfer Protocol (HTTP), where the POST request includes the image to be processed.

7. A display test method (400) for an HMI, characterized in that: Applied to a second electronic device (102), the method comprises: Receiving an image to be processed sent by a first electronic device (101); the image to be processed is an image obtained by photographing a human-machine interface HMI device (103) by a camera (104) when the first electronic device controls the HMI device to display a preset element; Obtaining recognition data of the image to be processed using a pre-trained element recognition model according to the image to be processed; the recognition data includes information of a target element in the image to be processed; The identification data is sent to the first electronic device.

8. The method according to claim 7, characterized in that The obtaining recognition data of the image to be processed by using a pre-trained element recognition model according to the image to be processed includes: Obtaining the position of a reference frame in the image to be processed; Input the position of the image to be processed and the reference frame into the element recognition model, and output a text recognition result and / or an image recognition result; the text recognition result includes the value, position and confidence of the target element of the text type in the image to be processed, and the image recognition result includes the value, position and confidence of the target element of the image type in the image to be processed; The recognition data in json data format is generated according to the text recognition result and / or the image recognition result.

9. The method according to claim 8, characterized in that The element recognition model includes a text recognition model and an image recognition model; The step of inputting the position of the image to be processed and the reference frame into the element recognition model and outputting a text recognition result and / or an image recognition result comprises: Determining the type of target element in the image to be processed; If the type of the target element includes a text type, inputting the position of the image to be processed and the reference frame into the text recognition model, and outputting the text recognition result; If the type of the target element includes an image type, the position of the image to be processed and the reference frame are input into the image recognition model, and the image recognition result is output.

10. The method according to any one of claims 7 to 9, characterized in that: The receiving the image to be processed sent by the first electronic device includes: A POST request sent by the first electronic device is received according to the Hypertext Transfer Protocol (HTTP), where the POST request includes the image to be processed.

11. A display test device (500) for an HMI, characterized in that: include: A control module (501) is used to start the automated test and control the human-machine interface (HMI) device to display preset elements; A first acquisition module (502), used to acquire an image to be processed obtained by photographing the HMI device with a camera; A second acquisition module (503) is used to acquire recognition data of the image to be processed; wherein the recognition data is obtained according to the image to be processed using a pre-trained element recognition model, and the recognition data includes information of a target element in the image to be processed; The verification module (504) is used to compare and verify the information of the target element and the information of the preset element to obtain the display test result of the HMI device.

12. A display test device (600) for an HMI, characterized in that: include: A receiving module (601), used for receiving an image to be processed sent by a first electronic device; The image to be processed is an image obtained by photographing the human-machine interface HMI device by a camera when the first electronic device controls the HMI device to display a preset element; A recognition module (602) is used to obtain recognition data of the image to be processed using a pre-trained element recognition model according to the image to be processed; the recognition data includes information of a target element in the image to be processed; The sending module (603) is used to send the identification data to the first electronic device.

13. An electronic device (700), comprising: A processor (702), a communication interface (704), a memory (706) and a bus (708), wherein the processor (702), the communication interface (704) and the memory (706) communicate with each other via the bus (708); The memory (706) is used to store at least one executable instruction, and the executable instruction enables the processor (702) to perform an operation corresponding to the method as described in any one of claims 1 to 6, or to perform an operation corresponding to the method as described in any one of claims 7 to 10.

14. A computer-readable storage medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6, or the method according to any one of claims 7 to 10.

15. A computer program product, the computer program product being tangibly stored on a computer readable medium and comprising computer executable instructions, which when executed cause at least one processor to perform the method according to any one of claims 1-6, or to perform the method according to any one of claims 7-10.

Images