Intelligent cabin performance detection method, device, equipment and medium

Abstract

The invention relates to the technical field of intelligent vehicles, and provides an intelligent cabin performance detection method and device, equipment and a medium. According to the method, firstly, connection between a test device and a to-be-detected intelligent cabin is established, then the test device sends a target operation event execution message to the intelligent cabin, and a target detection model is started to detect an interface of the intelligent cabin; and finally, according to the target interface detection result returned by the target detection model, determining the performance detection result of the intelligent cabin in response to the target operation event, and performing interface identification based on the intelligent detection model to automatically complete the "smoking test" of the intelligent cabin without manual operation, thereby improving the detection efficiency, reducing the detection cost, and improving the detection efficiency. And the consistency of detection results is good.

Description

Technical Field

[0001] The present application relates to the field of smart vehicle technology, and in particular to a smart cockpit performance detection method, device, equipment and medium. Background Art

[0002] With the development of smart cockpits, they provide rich graphical interfaces and display technologies, providing users with a wide variety of entertainment and information services. Starting the vehicle is the most critical step for users to use the vehicle. When the user starts the vehicle, abnormalities in the smart cockpit interface will affect the user experience. Therefore, when starting the vehicle, you can first perform a "smoke test" on the smart cockpit to detect whether the smart cockpit interface responds to various events normally.

[0003] How to conduct a "smoke test" quickly and accurately to improve the overall testing efficiency of the smart cockpit is a technical problem that needs to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a smart cockpit performance detection method, device, equipment and medium to solve the problem of how to quickly and accurately perform a "smoke test" on the smart cockpit in the prior art.

[0005] In a first aspect of an embodiment of the present application, a method for detecting performance of a smart cockpit based on interface recognition is provided. The method is executed by a test device and is used to detect the performance of a smart cockpit in response to a target operation event. The method comprises:

[0006] Establishing a connection with the smart cockpit to be detected, where the connection at least includes a communication connection;

[0007] Sending a target operation event message to the smart cockpit;

[0008] Start the target detection model to detect the smart cockpit interface;

[0009] In response to receiving the target interface detection result sent by the target detection model, a performance detection result of the smart cockpit responding to the target operation event is generated based at least on the target interface detection result.

[0010] According to a second aspect of an embodiment of the present application, a smart cockpit performance detection device based on interface recognition is provided, comprising:

[0011] An establishing module is configured to establish a connection with the smart cockpit to be detected, where the connection at least includes a communication connection;

[0012] A sending module, configured to send an execution target operation event message to the smart cockpit;

[0013] A startup module is configured to start a target detection model to detect the smart cockpit interface;

[0014] The detection module is configured to generate a performance detection result of the smart cockpit responding to the target operation event based at least on the target interface detection result received from the target detection model.

[0015] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0016] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0017] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application first establish a connection between the test device and the smart cockpit to be tested, then the test device sends a target operation event message to the smart cockpit, and starts the target detection model to detect the smart cockpit interface, and finally determines the performance test result of the smart cockpit in response to the target operation event based on the target interface detection result returned by the target detection model. The "smoke test" of the smart cockpit can be automatically completed by performing interface recognition based on the intelligent detection model without manual operation, thereby improving the detection efficiency, reducing the detection cost, and achieving good consistency in the detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flow chart of a smart cockpit performance detection method based on interface recognition provided in an embodiment of the present application.

[0020] Figure 2 It is a flow chart of another intelligent cockpit performance detection method based on interface recognition provided in an embodiment of the present application.

[0021] Figure 3 It is a flowchart of a method for obtaining smart cockpit interface video by a video thread provided in an embodiment of the present application.

[0022] Figure 4 It is a flowchart of a method for processing threads to detect smart cockpit interface video provided in an embodiment of the present application.

[0023] Figure 5 It is a flowchart of a method for transmitting a smart cockpit interface video to a test device by a display thread provided in an embodiment of the present application.

[0024] Figure 6 It is a schematic diagram of the overall process of the intelligent cockpit performance detection method based on interface recognition provided in an embodiment of the present application.

[0025] Figure 7 It is a flow chart of another intelligent cockpit performance detection method based on interface recognition provided in an embodiment of the present application.

[0026] Figure 8 It is a flowchart of reasoning and data processing of the target detection model provided in the embodiment of the present application.

[0027] Fig. 9 It is a schematic diagram of an intelligent cockpit performance detection device based on interface recognition provided in an embodiment of the present application.

[0028] Fig.10 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0030] A method and device for detecting performance of a smart cockpit based on interface recognition according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0031] As mentioned above, when the car computer is started, the smart cockpit can first be subjected to a "smoke test" to detect whether the smart cockpit interface responds to various events normally. In related technologies, the way to detect the smart cockpit after the car computer is started is usually manual visual recognition, which is inefficient when performing a large number of test tasks. At the same time, if an abnormality is detected, the car computer version may need to be revised and released, and then tested again. This series of processes is time-consuming and prone to misdetection.

[0032] That is, the detection methods in the related art have high labor costs, low detection efficiency and detection accuracy, and poor versatility. The detection results obtained by different inspectors may vary greatly. Therefore, it is necessary to provide a method that can quickly and accurately perform a "smoke test" to improve the overall testing efficiency of the smart cockpit.

[0033] In view of this, an embodiment of the present application provides a smart cockpit performance detection method based on interface recognition, by first establishing a connection between a test device and a smart cockpit to be detected, and then the test device sends a target operation event message to the smart cockpit, and starts a target detection model to detect the smart cockpit interface, and finally determines the performance detection result of the smart cockpit in response to the target operation event based on the target interface detection result returned by the target detection model. Interface recognition based on the intelligent detection model can automatically complete the "smoke test" of the smart cockpit without manual operation, thereby improving detection efficiency, reducing detection costs, and achieving good consistency in detection results.

[0034] Figure 1 FIG. 1 is a flow chart of a method for detecting performance of an intelligent cockpit based on interface recognition provided by an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0035] In step S101, a connection is established with the smart cockpit to be detected.

[0036] The connection at least includes a communication connection.

[0037] In step S102, a target operation execution event message is sent to the smart cockpit.

[0038] In step S103, the target detection model is started to detect the smart cockpit interface.

[0039] In step S104, in response to receiving the target interface detection result sent by the target detection model, a performance detection result of the smart cockpit responding to the target operation event is generated based at least on the target interface detection result.

[0040] In certain embodiments of the present application, the method may be performed by a test device. The test device may be, for example, a test terminal, which may have communication capabilities and be able to communicate with the smart cockpit to be tested to send test-related data to the smart cockpit, or receive relevant data sent by the smart cockpit or other auxiliary devices. Furthermore, the test device may also run a test program, through which the smart cockpit or other auxiliary devices are controlled to execute target events, and obtain various response data when the smart cockpit executes the target event, and analyze and process the obtained response data to obtain test results.

[0041] In some embodiments of the present application, the test device may first establish a connection with the smart cockpit to be tested. The connection includes at least a communication connection. For example, the test device may use a test program to obtain the device serial communication port (Communication Port, COM) of the smart cockpit, bind the COM port to the control device of the test program, and thereby establish a communication connection between the test device and the smart cockpit. In this case, the communication connection mode is serial communication. It is understandable that the test device may also establish other types of communication connections with the smart cockpit in other ways, which are not limited here.

[0042] Furthermore, the test equipment and the smart cockpit can also establish a physical connection so that the test equipment can control the smart cockpit to complete physical operations such as turning on the vehicle computer. For example, the vehicle computer power line of the smart cockpit can be connected to the relay device, and the relay device is also connected to the test equipment. The test equipment can control the relay device to close or open through the test program. When the relay device is closed, the vehicle computer of the smart cockpit is powered on and performs a power-on operation. Conversely, when the relay device is disconnected, the vehicle computer of the smart cockpit is powered off and performs a shutdown operation.

[0043] In some implementations, after establishing a connection with the smart cockpit, the test device may send a target operation event message to the smart cockpit. The target event may be a vehicle computer startup event in the smart cockpit, or a startup event of each application in the smart cockpit after the vehicle computer is powered on, or an interface switching event corresponding to each display screen in the smart cockpit after the vehicle computer is powered on, such as a Universal Serial Bus (USB) plug-in event, a 360-degree panoramic interface loading, etc.

[0044] On the other hand, the test equipment can also start the target detection model to detect the smart cockpit interface. Among them, the target detection model can be a preset model, which can be deployed locally in the smart cockpit, or deployed in a camera device for shooting the display screen of the smart cockpit interface, or deployed in the cloud, without limitation here. Furthermore, the target detection model can detect the display screen when the smart cockpit performs a target operation event to obtain the target interface detection result, and send the target interface detection result to the test equipment.

[0045] Furthermore, the test device may receive a target interface detection result sent by the target detection model, and generate a performance detection result of the smart cockpit in response to a target operation event based at least on the target interface detection result.

[0046] According to the technical solution provided in the embodiment of the present application, a connection is first established between the test equipment and the smart cockpit to be tested, and then the test equipment sends a target operation event message to the smart cockpit, and starts the target detection model to detect the smart cockpit interface. Finally, the performance test result of the smart cockpit in response to the target operation event is determined based on the target interface detection result returned by the target detection model. The "smoke test" of the smart cockpit can be automatically completed by performing interface recognition based on the intelligent detection model without manual operation, thereby improving detection efficiency, reducing detection costs, and achieving good consistency in detection results.

[0047] In some embodiments of the present application, the target interface detection may include successful execution and failed execution. Successful execution means that the target detection algorithm successfully detects the target interface within a preset time, and failed execution means that the target detection algorithm does not successfully detect the target interface within a preset time. Furthermore, the target interface is the interface displayed by the smart cockpit after the target operation event is successfully executed.

[0048] In other words, the target detection algorithm can detect the display screen when the smart cockpit executes the target operation event, and determine whether the interface in the display screen matches the target interface. If so, it means that the smart cockpit can respond to the target operation event normally, and then generate corresponding detection results as proof that the smart cockpit has passed the "smoke test".

[0049] Furthermore, in order to improve the detection efficiency, the detection time can also be set. For example, a preset duration can be set, and the value of the preset duration can be determined according to the smart cockpit response time range corresponding to different target operation events specified by business requirements. If the target detection algorithm detects a real-time interface that matches the target interface within the preset duration, a successful target interface detection result is generated and sent to the test device. Conversely, if the target detection algorithm does not detect a real-time interface that matches the target interface within the preset duration, a failed target interface detection result is generated and sent to the test device.

[0050] Alternatively, in some embodiments, if the target detection algorithm fails to detect a real-time interface that matches the target interface within a preset time range, the target interface detection result may not be sent to the test device. The test device itself determines that the smart cockpit has failed to execute the target operation event after determining the timeout, and uses the execution failure as the target interface detection result, and generates a performance test result of the smart cockpit responding to the target event based on this.

[0051] In certain embodiments of the present application, if the test device determines that the target interface detection result is a successful execution, the performance detection result can be determined to be a successful response. Conversely, if the test device determines that the target interface detection result is a failed execution, the performance detection result can be determined to be a failed response. Furthermore, the test device can also generate a test report and save corresponding logs and videos when the response fails for the user to view and analyze.

[0052] In this way, the "smoke test" result of whether the cockpit responds to the target operation event successfully can be achieved conveniently and accurately, providing reliable "smoke test" credentials for subsequent more in-depth performance tests.

[0053] In some implementations of the present application, the performance detection result may also include response time.

[0054] Figure 2 is a flow chart of another intelligent cockpit performance detection method based on interface recognition provided by an embodiment of the present application. Figure 2 Steps S201 to S202 in the embodiment shown are Figure 1 Steps S101 to S102 in the illustrated embodiment are substantially the same and will not be described in detail herein. Figure 2 As shown, the method also includes the following steps:

[0055] In step S203, in response to receiving a target operation event start success response message sent by the smart cockpit, it is determined that the timestamp of the start success response message is a first timestamp.

[0056] In step S204, in response to receiving the target interface detection result sent by the target detection model, the timestamp of the target interface detection result is determined to be a second timestamp.

[0057] In step S205, a target interface detection result sent by the target detection model is received, and in response to determining that the target interface detection result is successfully executed, a difference between the second timestamp and the first timestamp is determined as the response time of the smart cockpit in response to the target operation event.

[0058] In some embodiments of the present application, when the test device sends an execution target operation event message to the smart cockpit, the smart cockpit can initiate a target operation event according to the received execution target operation event message, and send a target operation event initiation success response message to the test device after the initiation is successful. The test device receives the target operation event initiation success response message and uses the timestamp therein as the first timestamp.

[0059] For example, if the target operation event is the vehicle computer power-on event of the smart cockpit, the test device can send a control signal to the relay to control its closing, thereby controlling the vehicle computer to power on. After the vehicle computer is powered on, it can send a power-on response message to the test device through the COM port. The power-on response message carries a timestamp, and the test device can record the timestamp as the first timestamp.

[0060] For another example, if the target operation event is a music application startup event in the smart cockpit, the test device can send a music application startup message to the smart cockpit, and the smart cockpit executes the music application startup operation after receiving the message, and returns a music application startup response message to the test device. The music application startup response message also carries a timestamp, and the test device can also record the timestamp as the first timestamp.

[0061] Furthermore, when the target interface detection result is successful execution, the test device can also determine that the timestamp in the received target interface detection result is the second timestamp. That is, after the target detection model successfully detects the real-time interface that matches the target interface, a target interface detection result including execution success information and the current timestamp is generated and sent to the test device. The test device receives the target interface detection result and records the timestamp therein as the second timestamp.

[0062] For example, if the target operation event is a USB plug-in and unplug-out event in the smart cockpit, the target detection model can generate and send a target interface detection result to the test device when it detects a real-time interface that matches the target interface corresponding to the USB plug-in and unplug-out event. When the test device receives the target interface detection result, it can first determine whether it contains execution success information. If so, the timestamp of the target interface detection result is recorded as the second timestamp.

[0063] Alternatively, if the target detection model only sends the target interface detection result to the test device when the execution is successful, and the test device automatically determines that the target detection execution has failed after a timeout, the test device can directly record the timestamp of the target interface detection result as the second timestamp when receiving the target interface detection result.

[0064] In this way, the response time of the smart cockpit in response to the target operation event can be detected conveniently and accurately, which can be used as a reference factor for judging the performance of the smart cockpit in responding to the target operation event, thereby improving the overall test efficiency.

[0065] In some embodiments of the present application, starting the target detection model to detect the smart cockpit interface may include sending a first start message to a video recording application to enable the video recording application to record the smart cockpit interface video. Also, sending a second start message to an inference device to enable the inference device to start a video thread to obtain the smart cockpit interface video, start a processing thread to call the target detection model to detect the smart cockpit interface video, and start a display thread to transmit the smart cockpit interface video to a test device for display.

[0066] In some implementations, the second startup message may carry target operation event information. The target operation event information may include a target operation event identifier, and the processing thread may obtain the target interface corresponding to the target operation event locally or in the cloud based on the target operation event identifier. Alternatively, the target operation event information may also include a target interface corresponding to the target operation event, and the processing thread may directly use the target interface for detection.

[0067] That is to say, when the test device starts the target detection model, on the one hand, it needs to start the video recording application to record the video of the smart cockpit interface, and on the other hand, it needs to call the target detection model to detect the recorded video to obtain the target interface detection result. Among them, the video recording application can be a built-in screen recording application in the smart cockpit, or it can also be a video recording application corresponding to an external camera device.

[0068] The target detection model can be configured in an inference device, which can be the smart cockpit itself, or a camera device, or other devices that can be connected to the smart cockpit or an external camera device. Furthermore, the inference device is also connected to a test device to send the target interface detection result to the test device.

[0069] After receiving the message to start the target detection model, the inference device can start the video thread, processing thread and display thread to obtain and preprocess the video of the smart cockpit respectively, perform inference processing on the preprocessed video, and send the inference processed video to the test device for display.

[0070] Figure 3 1 is a flow chart of a method for obtaining a video of a smart cockpit interface by a video thread provided in an embodiment of the present application. Figure 3 As shown, the method comprises the following steps:

[0071] In step S301, a smart cockpit interface video is obtained, and a current image frame in the smart cockpit interface video is converted into an initialized instance object.

[0072] In step S302, the initialization instance object is converted into an initialization production queue.

[0073] In step S303, all image frames in the smart cockpit interface video are processed in a loop to obtain a production queue of the smart cockpit interface video.

[0074] In some embodiments of the present application, the video thread can obtain a video of the real-time display interface of the smart cockpit screen recorded from the smart cockpit or an external camera device, and convert the current image frame in the video into an initialized instance object. For example, if the instance object of each image frame in the video is represented by the parameter cap, the cap can be initialized to obtain an initialized production queue. Furthermore, the production queue of the smart cockpit interface video can be obtained by sequentially combining the initialized production queues obtained after the initialization operation of the instance objects of all image frames, for example, recorded as pride_q.

[0075] That is to say, the video thread can transmit the video of the smart cockpit interface frame by frame. When transmitting, it first initializes a method object to open the video stream to facilitate the test program to process the call; then uses the queue to store each image frame of the video stream for easy processing; finally, it loops to produce each frame and adds each frame to the production queue to obtain pride_q.

[0076] Figure 4 1 is a flow chart of a method for processing a thread to detect a video of an intelligent cockpit interface provided by an embodiment of the present application. Figure 4 As shown, the method comprises the following steps:

[0077] In step S401, a target interface is determined according to the target operation event information, where the target interface is the interface displayed by the smart cockpit after the target operation event is successfully executed.

[0078] In step S402, each image frame is sequentially acquired from the production queue, and the confidence level of the current image frame matching the target interface is calculated.

[0079] In step S403, in response to the confidence level of the target image frame and the target interface being matched within the preset time period being greater than a preset threshold, the target interface detection result is determined to be a successful execution.

[0080] The target operation event includes one or more operation events, and each target operation event corresponds to one or more target interfaces.

[0081] In certain embodiments of the present application, the processing thread can determine the target interface based on the target operation event information, and the target interface is the interface displayed by the smart cockpit after the target operation event is successfully executed. For example, if the target operation event is the vehicle startup event of the smart cockpit, the target interface can be the vehicle startup interface of the smart cockpit. Alternatively, if the target operation event is the startup event of the navigation application in the smart cockpit, the target interface can be the navigation application startup interface. Alternatively, if the target operation event is the USB insertion event, the target interface can be an interface prompting a successful USB connection.

[0082] In some implementations, the processing thread may call a target detection model, and input a determined target interface into the target detection model as a detection target. The detection target may be one or more. For example, the target interface input into the target detection model may be an interface corresponding to the same target operation event, or multiple interfaces, such as a successful startup interface of a target application startup event, or a welcome interface and a successful startup interface of a target application startup event. Alternatively, the target interface input into the target detection model may also be multiple interfaces corresponding to multiple target operation events, such as a successful startup interface of the first to Nth application startup events and a successful switching interface of the second to Mth application page switching events.

[0083] On the other hand, the processing thread can obtain the production queue produced by the video thread, obtain each image frame from the production queue in turn, and calculate the confidence that the current image frame matches the target interface. If the processing thread detects a target image frame with a confidence level greater than a preset threshold that matches the target interface within a preset time, the target interface detection result can be considered to be executed successfully, and then the successful target interface detection result is sent to the test device. Among them, the preset threshold of the confidence level can be set according to actual needs, for example, set to 0.99.

[0084] That is to say, the processing thread can obtain the production queue data, so as to add each frame to the target detection model, and then obtain the data in the production queue frame by frame. The obtained data is input into the target detection model frame by frame, and the obtained target interface is input into the target detection model, and then the recognition judgment result of the target detection model is obtained, where the target interface, i.e., the detection target of the model, can be a single target or multiple targets. If the confidence between the current data frame and the target interface is less than 0.99, the target detection model can output the detection status as false (false), otherwise if the confidence is greater than 0.99, the target detection model can output the detection status as true (true).

[0085] Figure 5 1 is a flow chart of a method for transmitting a video of a smart cockpit interface to a test device by a display thread provided in an embodiment of the present application. Figure 5 As shown, the method comprises the following steps:

[0086] In step S501, the image frames currently processed by the processing thread are added to the display queue in sequence.

[0087] In step S502, the presentation queue is transmitted to the test device so that the test device reassembles the image frames in the presentation queue to display the smart cockpit interface video.

[0088] In certain embodiments of the present application, the presentation thread may sequentially add the image frames currently processed by the processing thread to a presentation queue, and then transmit the presentation queue to a test device so that the test device reassembles the image frames in the presentation queue to display the smart cockpit interface video.

[0089] That is, the presentation thread may add the image frames processed by the processing thread to the presentation queue for interface monitoring.

[0090] In some embodiments of the present application, the target detection model can be a trained YOLOv5 model. During model training, various historical data in the smart cockpit can be obtained as training samples, such as the startup screen of the smart cockpit central control, the interfaces of various applications in the smart cockpit central control, such as the music application interface, the settings application interface, etc., and the interfaces of other custom scenes in the smart cockpit, such as 360 panoramic pictures, etc., as training samples. Furthermore, the startup screen of the smart cockpit instrument and the interface of the custom scene of the smart cockpit instrument can also be obtained as training samples. Sample labels can be added to each type of training sample.

[0091] Among them, the startup screen of the smart cockpit central control can add the characteristic picture or icon of the central control screen startup screen as a sample description, the interface of each application in the smart cockpit central control can add the characteristic picture or icon of each application interface as a sample description, the interface of other custom scenes in the smart cockpit can add the characteristic picture or icon of each custom scene interface as a sample description, the startup screen of the smart cockpit instrument can add the characteristic picture or icon of the instrument screen startup screen as a sample description, and the custom scene of the smart cockpit instrument can add the interface characteristic picture or icon of each custom scene theme as a sample description.

[0092] In one implementation, during model training, 70% of the samples can be used as a training set, 20% as a validation set, and 10% as a test set, or other proportions can be used to divide the training set, validation set, and test set. The training effect evaluation can refer to the precision and recall curve, or refer to the average precision, confusion matrix, etc., which is not limited here.

[0093] Figure 6 FIG. 1 is a schematic diagram of the overall process of the intelligent cockpit performance detection method based on interface recognition provided by the embodiment of the present application. Figure 6As shown in the figure, the overall process includes the power-on phenomenon detection process and the model training and reasoning process. In the power-on phenomenon detection process, the smart cockpit and the test equipment can be connected by a relay device, and then the vehicle startup of the smart cockpit can be simulated through program control in the test equipment, and then the power-on phenomenon detection is performed. The detection results are compared with the expected results and the results are reported.

[0094] Among them, an artificial intelligence (AI) model can be called to detect power-on phenomena, such as target detection models such as YOLOv5. The training and reasoning process of the model is to first obtain a set of image resources, annotate the images, and then perform model training. During training, a pre-trained model is first constructed, and then the trained model is obtained by training the annotated image resource set. Next, the interface video stream when the smart cockpit is turned on is obtained, and the trained model is used to perform image reasoning and data processing on the video stream, and the processing result is used as the power-on phenomenon detection result.

[0095] In other words, the power-on phenomenon detection process includes circuit connectivity, program control, simulated vehicle startup, power-on phenomenon detection, comparison with expected results, and reporting results. The model training and pre-inference process mainly includes image resource sets, annotations, model training, pre-trained models, trained models, video streams, image inference, and data processing.

[0096] When the cockpit is powered on and the vehicle is powered on, the power cord can be controlled by the program to control the vehicle's power on / off status. The AI ​​model can then be used to detect various phenomena after the vehicle is powered on, including power on response time and cockpit application program interface verification. The test results are obtained by processing the real-time monitoring data and generating a report, which can be quickly implemented in multiple scenarios such as the actual smart cockpit "smoke test", power on response time, and vehicle startup special detection tasks.

[0097] Figure 7 FIG. 1 is a flow chart of another method for detecting performance of an intelligent cockpit based on interface recognition provided by an embodiment of the present application. Figure 7 As shown, first connect the power cord to the relay device, connect the relay device to the test equipment, then obtain the COM port of the relay device, and establish communication between the test equipment and the relay device. Next, you can turn on the AI ​​model detection, and at the same time, the test equipment controls the switch of the relay device to close through the test program, so that the power supply of the vehicle computer is connected, thereby starting the vehicle computer. After the test program calls the relay device vehicle computer startup detection process, the test equipment obtains and records the current timestamp as the start timestamp.

[0098] When the power cord of the vehicle computer is connected, the vehicle computer automatically starts and the smart cockpit enters the power-on interface. The target detection model can brake and detect the target interface of the smart cockpit. These target interfaces can be input into the target detection model according to the detection needs, including but not limited to the first frame of the smart cockpit power-on interface, such as instrument power-on, central control power-on, etc., and the target interface can be a multi-screen scene. When the target detection model detects the power-on interface, the timestamp can be obtained again as the end timestamp. The time difference between the end timestamp and the start timestamp is calculated to obtain the vehicle computer startup response time.

[0099] Furthermore, the test equipment can also control the startup of applications in the smart cockpit through the test program, for example, by means of adb commands or uiautomator2 commands. If using adb commands, the control instruction is, for example, adb shell am start-n "app package name". After starting the application, you can enter the interface of each application. The target detection model can use a similar method to detect again whether the interface of each instrument or other screen includes the corresponding interface of each application. If it is included, the target detection model outputs true, then the detection passes. Otherwise, if the target detection model outputs false, then the detection fails, indicating that some applications may have abnormalities, and it is necessary to retain videos and logs for users to view and analyze. Finally, record the test results and generate a report.

[0100] Figure 8 Schematic diagram of the process of reasoning and data processing of the target detection model provided in the embodiment of the present application. Figure 8 As shown, when the target detection model performs reasoning and data processing, it first performs the video stream frame-by-frame transmission processing, the video stream frame-by-frame processing process, the model loading and detection reasoning process, and the video stream display process.

[0101] When transmitting the video stream frame by frame, first initialize a video stream method object, namely the cap object, to facilitate the test program to process the call, and then use the queue to store each image frame of the video stream to initialize the production queue. Finally, loop to produce each frame, add each frame to the production queue, and obtain the production queue produce_q.

[0102] When performing the frame-by-frame processing of the video stream, you can first obtain the production queue data to facilitate adding each frame to the target detection model. Then you can use the parameter get_frame_byte to obtain the queue data for each frame. Next, call the target detection model, such as the YOLOv5 model for inference, to obtain the inferred json data. The command to call the target detection model can be, for example, res_json = ai_to_json_post_frame(frame). In an example, the format of the inferred json data can be: {'result': [{'xmin': 504, 'ymin': 205, 'xmax': 516, 'ymax': 218, 'confidence': 0.99, 'class': 21, 'name': 'Level 1 airbag failure'}]}, where xmin, ymin, xmax and ymax are all coordinate values ​​of the image frame, confidence is the confidence, class is the classification label, and name is the name.

[0103] Next, you can fill in the detection target, that is, the target interface, and then perform detection and recognition judgment. The detection target can be a single target or multiple targets. Among them, the detection and recognition judgment instruction can be, for example, Check_by_frame(labels), where labels is the detection target. The confidence of each image frame and the detection target is detected, that is, the probability that each image frame is the same as the detection target. If the confidence is greater than a threshold value, such as 0.99, the detection status is true, otherwise the detection status is false. Finally, the processed frame can be added to the display queue display_q for the interface monitoring detection process.

[0104] When loading the model and performing detection inference, you can first load the target detection model. If the model loading is abnormal, reinitialize it. After successful loading, the data in the production queue is passed frame by frame, and the target detection model is used for recognition.

[0105] When displaying a video stream, you can loop through the display queue display_q to get data frame by frame, then display the frame as an image. The loop display is the video display, and finally save the image and video.

[0106] By adopting the technical solution of the embodiment of the present application, a target detection model is combined with a test program to control the automatic process of vehicle computer startup. During the operation, a program control process is performed to simulate the actual detection process, which reduces the detection accuracy problems caused by human operation and forms a complete solution for vehicle computer startup to smart cockpit phenomenon detection, greatly improving the detection efficiency.

[0107] The technical solution provided by the embodiment of the present application is simple to operate. It only needs to run the program to automatically detect the smart cockpit phenomenon after the car computer is started and generate a report, which greatly reduces the human resource cost. In addition, the solution controls the startup of the car computer through the program, and then starts each application or other customized scenarios. It has a good general effect, can be fast and real-time in the smart cockpit related detection links, improve the generation efficiency, and can be used as a reference for version smoke testing.

[0108] The technical solution provided in the embodiment of the present application can automatically judge the expected results during the intelligent cockpit phenomenon detection process after the vehicle computer is started, automatically generate a customized detection report, and record the detection process video in real time.

[0109] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0110] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0111] Fig. 9 is a schematic diagram of an intelligent cockpit performance detection device based on interface recognition provided by an embodiment of the present application. Fig. 9 As shown, the device comprises:

[0112] The establishing module 901 is configured to establish a connection with the smart cockpit to be detected, and the connection at least includes a communication connection.

[0113] The sending module 902 is configured to send an execution target operation event message to the smart cockpit.

[0114] The startup module 903 is configured to start the target detection model to detect the smart cockpit interface.

[0115] The detection module 904 is configured to generate a performance detection result of the smart cockpit responding to the target operation event based at least on the target interface detection result received from the target detection model.

[0116] According to the technical solution provided in the embodiment of the present application, a connection is first established between the test equipment and the smart cockpit to be tested, and then the test equipment sends a target operation event message to the smart cockpit, and starts the target detection model to detect the smart cockpit interface. Finally, the performance test result of the smart cockpit in response to the target operation event is determined based on the target interface detection result returned by the target detection model. The "smoke test" of the smart cockpit can be automatically completed by performing interface recognition based on the intelligent detection model without manual operation, thereby improving detection efficiency, reducing detection costs, and achieving good consistency in detection results.

[0117] In some embodiments, the target interface detection results include successful execution and failed execution. Successful execution means that the target detection algorithm successfully detects the target interface within a preset time period, and failed execution means that the target detection algorithm does not successfully detect the target interface within the preset time period. The target interface is the interface displayed by the smart cockpit after the target operation event is successfully executed; the performance detection result of the smart cockpit in response to the target operation event is generated at least based on the target interface detection result, including: in response to determining that the target interface detection result is a successful execution, determining the performance detection result is a successful response; in response to determining that the target interface detection result is a failed execution, determining the performance detection result is a failed response.

[0118] In some embodiments, the performance detection result also includes response time; after sending a target operation event execution message to the smart cockpit, the method also includes: in response to receiving a target operation event start success response message sent by the smart cockpit, determining that the timestamp of the start success response message is a first timestamp; in response to receiving a target interface detection result sent by a target detection model, determining that the timestamp of the target interface detection result is a second timestamp; in response to the target interface detection result being a successful execution, generating a performance detection result of the smart cockpit responding to the target operation event based at least on the target interface detection result, including: determining that the difference between the second timestamp and the first timestamp is the response time of the smart cockpit responding to the target operation event.

[0119] In some embodiments, starting a target detection model to detect the smart cockpit interface includes: sending a first startup message to a video recording application to enable the video recording application to record the smart cockpit interface video; sending a second startup message to an inference device to enable the inference device to start a video thread to obtain the smart cockpit interface video, start a processing thread to call the target detection model to detect the smart cockpit interface video, and start a display thread to transmit the smart cockpit interface video to a test device for display.

[0120] In some embodiments, the video thread obtains the smart cockpit interface video in the following manner: obtain the smart cockpit interface video, convert the current image frame in the smart cockpit interface video into an initialization instance object; convert the initialization instance object into an initialization production queue; and loop through all image frames in the smart cockpit interface video to obtain the production queue of the smart cockpit interface video.

[0121] In some embodiments, the second startup message carries target operation event information; the processing thread detects the smart cockpit interface video in the following manner: determining the target interface based on the target operation event information, the target interface being the interface displayed by the smart cockpit after the target operation event is successfully executed; obtaining each image frame in sequence from the production queue, and calculating the confidence that the current image frame matches the target interface; in response to the confidence that the target image frame matches the target interface detected within a preset time period is greater than a preset threshold, determining that the target interface detection result is a successful execution; wherein, the target operation event includes one or more operation events, and each target operation event corresponds to one or more target interfaces.

[0122] In some embodiments, the display thread transmits the smart cockpit interface video to the test device for display in the following manner: the image frames currently processed by the processing thread are added to the display queue in sequence; the display queue is transmitted to the test device so that the test device reorganizes the image frames in the display queue to display the smart cockpit interface video.

[0123] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0124] Fig.10 Schematic diagram of an electronic device provided in an embodiment of the present application. Fig.10 As shown, the electronic device 10 of this embodiment includes: a processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the computer program 1003, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1001 executes the computer program 1003, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0125] The electronic device 10 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 10 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art will appreciate that Fig.10 The electronic device 10 is merely an example and does not limit the electronic device 10 . The electronic device 10 may include more or fewer components than shown in the figure, or different components.

[0126] Processor 1001 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0127] The memory 1002 may be an internal storage unit of the electronic device 10, for example, a hard disk or memory of the electronic device 10. The memory 1002 may also be an external storage device of the electronic device 10, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 10. The memory 1002 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 1002 is used to store computer programs and other programs and data required by the electronic device.

[0128] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0129] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0130] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A smart cockpit performance detection method based on interface recognition, characterized in that: The method is performed by a test device to detect the performance of the smart cockpit in responding to a target operation event; The method comprises: Establishing a connection with the smart cockpit to be detected, wherein the connection at least includes a communication connection; Sending an execution target operation event message to the smart cockpit; Starting a target detection model to detect the smart cockpit interface; In response to receiving the target interface detection result sent by the target detection model, a performance detection result of the smart cockpit in response to the target operation event is generated at least based on the target interface detection result.

2. The method according to claim 1, characterized in that The target interface detection result includes execution success and execution failure. The execution success means that the target detection algorithm successfully detects the target interface within the preset time length. The execution failure means that the target detection algorithm does not successfully detect the target interface within the preset time length. The target interface is the interface displayed by the smart cockpit after the target operation event is successfully executed. The generating, at least based on the target interface detection result, a performance detection result of the smart cockpit in response to the target operation event includes: In response to determining that the target interface detection result is an execution success, determining that the performance detection result is a response success; In response to determining that the target interface detection result is an execution failure, the performance detection result is determined to be a response failure.

3. The method according to claim 2, characterized in that The performance test results also include response time; After sending the target operation execution event message to the smart cockpit, the method further includes: In response to receiving a target operation event start success response message sent by the smart cockpit, determining a timestamp of the start success response message as a first timestamp; In response to receiving the target interface detection result sent by the target detection model, determining the timestamp of the target interface detection result as a second timestamp; In response to the target interface detection result being successful, generating a performance detection result of the smart cockpit responding to the target operation event based at least on the target interface detection result, includes: The difference between the second timestamp and the first timestamp is determined as the response time of the smart cockpit in response to the target operation event.

4. The method according to claim 1, characterized in that The starting target detection model to detect the smart cockpit interface includes: Sending a first start message to a video recording application so that the video recording application records the smart cockpit interface video; A second startup message is sent to the inference device so that the inference device starts a video thread to obtain the smart cockpit interface video, starts a processing thread to call the target detection model to detect the smart cockpit interface video, and starts a display thread to transmit the smart cockpit interface video to the test device for display.

5. The method according to claim 4, characterized in that The video thread obtains the smart cockpit interface video in the following manner: Acquire the smart cockpit interface video, and convert the current image frame in the smart cockpit interface video into an initialized instance object; Convert the initialized instance object into an initialized production queue; All image frames in the smart cockpit interface video are processed in a loop to obtain a production queue of the smart cockpit interface video.

6. The method according to claim 5, characterized in that The second start message carries the target operation event information; The processing thread detects the smart cockpit interface video in the following manner: Determine a target interface according to the target operation event information, where the target interface is the interface displayed by the smart cockpit after the target operation event is successfully executed; Obtaining each image frame from the production queue in sequence, and calculating the confidence level that the current image frame matches the target interface; In response to the confidence level of the target image frame being detected to match the target interface within the preset time period being greater than a preset threshold, determining that the target interface detection result is a successful execution; The target operation event includes one or more operation events, and each target operation event corresponds to one or more target interfaces.

7. The method according to claim 6, characterized in that The display thread transmits the smart cockpit interface video to the test device for display in the following manner: Sequentially add the image frames currently processed by the processing thread to the display queue; The display queue is transmitted to the test device, so that the test device reassembles the image frames in the display queue to display the smart cockpit interface video.

8. An intelligent cockpit performance detection device based on interface recognition, characterized in that: include: An establishing module, configured to establish a connection with the smart cockpit to be detected, wherein the connection at least includes a communication connection; A sending module, configured to send an execution target operation event message to the smart cockpit; A startup module is configured to start a target detection model to detect the smart cockpit interface; The detection module is configured to generate a performance detection result of the smart cockpit responding to the target operation event based at least on the target interface detection result in response to receiving the target interface detection result sent by the target detection model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images