Intelligent cockpit instrument self-checking method and device and electronic equipment

Abstract

The invention relates to the technical field of intelligent vehicles, and provides an intelligent cabin instrument self-checking method and device and electronic equipment. The method comprises the following steps: starting a target detection model to detect the state of a target indicator light in an intelligent cabin under the condition of determining that the intelligent cabin starts instrument self-inspection, and obtaining a first time when the target indicator light is lighted for the first time and a second time when the target indicator light is extinguished for the first time after the target indicator light is lighted; when the absolute value of the time difference between the second time and the first time is smaller than the preset threshold value, it is determined that self-inspection of the instrument corresponding to the target indicator light passes, intelligent cabin instrument self-inspection verification based on interface detection is achieved, the accuracy of self-inspection result judgment is improved, and the judgment efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of smart vehicle technology, and in particular to a smart cockpit instrument self-checking method, device and electronic equipment. Background Art

[0002] With the development of smart cockpits, automotive instruments provide a rich graphical interface and display technology, which can help drivers understand the various parameters of the vehicle in a timely manner and ensure the normal operation and driving safety of the vehicle. The instrument indicator light self-check process provides the driver with key system information to ensure driving safety and serious safety. During the self-check process, the instrument will display various information of the vehicle. Through this information, the driver can better understand the vehicle's operating status and improve driving safety and comfort. Therefore, the phenomenon verification of the instrument self-check process is particularly important.

[0003] In the related art, self-checking and verification of instruments in smart cockpits is usually done manually by the naked eye, which is inefficient and prone to errors, and has poor versatility. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a method, device and electronic device for self-checking of intelligent cockpit instruments to solve the problems of low efficiency and poor versatility of self-checking of intelligent cockpit instruments in the prior art.

[0005] According to a first aspect of an embodiment of the present application, a smart cockpit instrument self-test method is provided, the method being executed by a test device and comprising:

[0006] In response to determining that the smart cockpit starts the instrument self-check, starting the target detection model to detect the state of the target indicator light in the smart cockpit, where the target indicator light is the self-check indicator light of any N instruments in the smart cockpit, where N is a positive integer;

[0007] Receive a first time and a second time returned by the target detection model, wherein the first time is the time when the target indicator light is first lit, and the second time is the time when the target indicator light is first extinguished after being lit;

[0008] Determine the time difference between the second time and the first time as the detection duration of the target indicator light;

[0009] In response to determining that the absolute value of the difference between the detection time and the preset display time is less than the preset threshold, it is determined that the instrument self-test corresponding to the target indicator light passes.

[0010] A second aspect of an embodiment of the present application provides an intelligent cockpit instrument self-checking device, comprising:

[0011] A starting module is configured to start a target detection model to detect a state of a target indicator light in the smart cockpit in response to determining that the smart cockpit starts the instrument self-check, where the target indicator light is a self-check indicator light of any N instruments in the smart cockpit, where N is a positive integer;

[0012] A receiving module is configured to receive a first time and a second time returned by the target detection model, wherein the first time is the time when the target indicator light is first lit, and the second time is the time when the target indicator light is first extinguished after being lit;

[0013] A determination module is configured to determine a time difference between the second time and the first time as a detection duration of the target indicator light;

[0014] The detection module is configured to determine that the instrument self-test corresponding to the target indicator light has passed in response to determining that the absolute value of the difference between the detection time and the preset display time is less than a preset threshold.

[0015] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0016] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application detect the status of the target indicator light in the smart cockpit by starting the target detection model under the condition that the smart cockpit instrument self-test is turned on, and obtain the first time when the target indicator light is first lit and the second time when it is first extinguished after lighting. When the absolute value of the time difference between the second time and the first time is less than a preset threshold, it is determined that the instrument self-test corresponding to the target indicator light has passed, thereby realizing smart cockpit instrument self-test verification based on interface detection, improving the accuracy of self-test result judgment, and improving judgment efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flowchart of an intelligent cockpit instrument self-check method provided in an embodiment of the present application.

[0019] Figure 2 It is a flow chart of a method for starting a target detection model to detect the status of a target indicator light in a smart cockpit provided in an embodiment of the present application.

[0020] Figure 3It is a flow chart of a method provided by an embodiment of the present application for calling a target detection model in a processing thread to detect video data of a target interface and obtaining the state of a target indicator light.

[0021] Figure 4 This is a flow chart of another method for starting a target detection model to detect the status of a target indicator light in a smart cockpit provided in an embodiment of the present application.

[0022] Figure 5 It is an overall flow chart of the self-check and verification of the intelligent cockpit instrument provided in the embodiment of the present application.

[0023] Figure 6 It is a structural diagram of the self-checking and verification system provided in an embodiment of the present application.

[0024] Figure 7 It is a workflow diagram of the video thread provided in the embodiment of the present application.

[0025] Figure 8 It is a workflow diagram of the processing thread provided in the embodiment of the present application.

[0026] Fig. 9 It is a schematic diagram of the workflow of the self-checking verification display method provided in an embodiment of the present application.

[0027] Fig.10 It is a flowchart of AI detection processing provided in an embodiment of the present application.

[0028] Fig.11 It is a flowchart of implementing instrument self-check verification and test result display based on the QT framework provided in an embodiment of the present application.

[0029] Fig.12 It is a detailed flow chart of the self-check verification of the intelligent cockpit instrument provided in the embodiment of the present application.

[0030] Fig.13 It is a schematic diagram of an intelligent cockpit instrument self-checking device provided in an embodiment of the present application.

[0031] Fig.14 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0033] A method and device for self-checking an intelligent cockpit instrument according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0034] As mentioned above, in the related art, the self-check verification of the instruments in the smart cockpit is usually completed by the naked eye. In the manual inspection process, the time for the instrument self-check indicator light to be continuously displayed and then extinguished is manually evaluated to determine whether the instrument self-check has passed. Table 1 shows the nominal continuous display time of some instrument indicator lights in the smart cockpit during self-check.

[0035] Table 1 Nominal display duration of some instrument indicators in the smart cockpit during self-check

[0036]

[0037]

[0038] When each instrument in the smart cockpit is self-checking, its indicator light needs to light up and go out after displaying the preset time continuously, indicating that the instrument self-check is normal. As shown in Table 1, the continuous display time of the indicator light is usually short, for example, 4 seconds. It is difficult for the human eye to accurately judge whether the actual continuous display time of the indicator light is 4 seconds. At the same time, when performing a large number of test tasks, the continuous display time of multiple indicator lights needs to be judged at the same time, which further increases the difficulty of manual detection.

[0039] That is to say, there is no solution in the relevant technology to accurately detect the continuous display duration of the indicator light during the self-test process. The existing detection methods are inefficient and prone to errors, and have poor versatility and high costs, resulting in the problem of abnormal continuous display duration of the instrument self-test indicator light being usually ignored.

[0040] In view of this, an embodiment of the present application provides a smart cockpit instrument self-check method, which detects the interface of the smart cockpit instrument during self-check through a target detection model, and records the first time when the indicator light of each instrument is first lit and the second time when it is first extinguished after lighting, so as to automatically detect the continuous display time of the indicator light of each instrument during self-check, and then judge whether the instrument self-check has passed based on the matching relationship between the continuous display time obtained by the detection and the nominal continuous display time of the indicator light, thereby realizing automatic verification of the instrument self-check.

[0041] Furthermore, the smart cockpit instrument self-check method provided in the embodiment of the present application detects the status of the target indicator light in the smart cockpit by starting the target detection model under the condition that the smart cockpit instrument self-check is turned on, and obtains the first time when the target indicator light is first lit and the second time when it is first extinguished after lighting. When the absolute value of the time difference between the second time and the first time is less than a preset threshold, it is determined that the instrument self-check corresponding to the target indicator light has passed, thereby realizing smart cockpit instrument self-check verification based on interface detection, improving the accuracy of self-check result judgment, and improving judgment efficiency.

[0042] Figure 1 is a flow chart of a method for self-checking an intelligent cockpit instrument provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0043] In step S101 , in response to determining that the smart cockpit starts the instrument self-check, a target detection model is started to detect the state of the target indicator light in the smart cockpit.

[0044] The target indicator light is the self-check indicator light of any N instruments in the smart cockpit, where N is a positive integer.

[0045] In step S102, the first time and the second time returned by the target detection model are received.

[0046] The first time is the time when the target indicator light is first lit, and the second time is the time when the target indicator light is first extinguished after being lit.

[0047] In step S103, the time difference between the second time and the first time is determined as the detection duration of the target indicator light.

[0048] In step S104, in response to determining that the absolute value of the difference between the detection time and the preset display time is less than the preset threshold, it is determined that the instrument self-check corresponding to the target indicator light has passed.

[0049] In some embodiments of the present application, the method may be performed by a test device, which may be, for example, a test terminal, which may have communication capabilities and can be connected to the smart cockpit to be tested to send test-related data to the smart cockpit, or receive relevant data sent by the smart cockpit or other auxiliary devices.

[0050] Furthermore, the test device can also run a test program to control the smart cockpit or other auxiliary devices to execute target events through the test program, and obtain various response data when the smart cockpit executes the target event, and analyze and process the obtained response data to obtain test results.

[0051] Furthermore, the testing device may also include a display interface, which may be used to display relevant information and test results during the testing process.

[0052] In some implementations, after determining that the smart cockpit has started the instrument self-test, the test device can start the target detection model to detect the status of the target indicator light in the smart cockpit. In one example, the smart cockpit can receive a test instruction sent by the test device and start the instrument self-test according to the test instruction. In another example, the smart cockpit can also start the instrument self-test by itself, which is not limited here.

[0053] The target detection model may be any AI (Artificial Intelligence) model capable of realizing target detection, such as the YOLOv5 model. The detection target of the target detection model may be the indicator light of each instrument. Furthermore, the detection target of the target detection model may be the indicator light of the lighting status of each instrument.

[0054] The target detection model detects the indicator light status of each instrument in the smart cockpit to obtain the first lighting time of each indicator light and record it as the first time of the indicator light. Furthermore, the target detection model can also obtain the first extinguishing time of each indicator light after the first lighting, and record it as the second time of the indicator light.

[0055] The test device receives the first time and the second time returned by the target detection model, and determines the time difference between the second time and the first time of the target indicator light as the detection time of the target indicator light. Further, the test device compares the detection time with the preset display time. If it is determined that the absolute value of the difference between the detection time and the preset display time is less than a preset threshold, it can be determined that the instrument self-test corresponding to the target indicator light has passed.

[0056] It is understandable that the number of target indicator lights can be one or more. That is, multiple instrument self-tests can be turned on at the same time in the smart cockpit. At this time, the target detection model uses the indicator lights of the lighting status of multiple instruments as detection targets, and records the first time and second time of each indicator light respectively. The test equipment determines whether the corresponding instrument self-test has passed based on the first time and second time of each indicator light received.

[0057] According to the technical solution provided in the embodiment of the present application, the target detection model is started to detect the status of the target indicator light in the smart cockpit under the condition that the instrument self-test of the smart cockpit is turned on, and the first time when the target indicator light is first lit and the second time when it is first extinguished after lighting are obtained. When the absolute value of the time difference between the second time and the first time is less than a preset threshold, it is determined that the instrument self-test corresponding to the target indicator light has passed, thereby realizing smart cockpit instrument self-test verification based on interface detection, improving the accuracy of self-test result judgment, and improving judgment efficiency.

[0058] Figure 2 1 is a flow chart of a method for starting a target detection model to detect the state of a target indicator light in a smart cockpit provided in an embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0059] In step S201, a start message is sent to a video recording application, so that the video recording application records the video of the target interface in real time.

[0060] The target interface at least includes a target indicator light.

[0061] In step S202, a processing thread is started, video data of a target interface is obtained in the processing thread, a target detection model is called to detect the video data of the target interface, and a state of a target indicator light is obtained.

[0062] In certain embodiments of the present application, after determining that the smart cockpit has started the instrument self-test, the test device may send a startup message to the video recording application, so that the video recording application can record the video of the target interface in real time. The video recording application may be a screen recording application built into the smart cockpit, or it may be a video recording application corresponding to an external camera device. The target interface includes at least an indicator light for the instrument that is self-testing. Furthermore, the target interface may also include indicator lights for instruments that need to be self-tested later.

[0063] Furthermore, the test device may also start a processing thread, obtain video data of the target interface in the processing thread, and call a target detection model to detect the video data of the target interface, thereby obtaining the state of the target indicator light.

[0064] Figure 3 This is a flow chart of a method for calling a target detection model in a processing thread to detect video data of a target interface and obtaining the state of a target indicator light, provided by an embodiment of the present application. Figure 3 As shown, the method comprises the following steps:

[0065] In step S301, the video data of the target interface is written into the first object of the processing thread.

[0066] In step S302, in response to receiving an on-target detection signal, a first object is instantiated.

[0067] In step S303, the target detection model is called to detect the confidence of each video data frame in the first object instance frame by frame.

[0068] In step S304, in response to determining that the confidence of the first target video data frame is greater than a preset confidence threshold, and the confidences of the preceding video data frames of the first target video data frame are less than or equal to the preset confidence threshold, the timestamp corresponding to the first target video data frame is determined to be the first time.

[0069] In step S305, in response to determining that the confidence of the second target video data frame is less than or equal to a preset confidence threshold, and the confidence of the previous video data frame of the second target video data frame is greater than the preset confidence threshold, the timestamp corresponding to the second target video data frame is determined to be the second time.

[0070] In some embodiments, the video data of the target interface may be first written into the first object of the processing thread, which may be the video parameter of the processing process, for example, denoted as cap1. After receiving the target detection signal, the processing thread may instantiate the first object to obtain a first object instance.

[0071] Furthermore, the target detection model can be called in the processing thread to detect the video data frames in the first object instance frame by frame and calculate the confidence of each video data frame, wherein the confidence of the video data frame refers to the confidence that the video data frame contains the detected target.

[0072] If the target detection model determines that the confidence of the first target video data frame is greater than the preset confidence threshold, and the confidence of the preceding video data frame of the first target video data frame is less than or equal to the preset confidence threshold, the timestamp corresponding to the first target video data frame is determined to be the first time. That is, when the target detection model detects that the target indicator light is lit for the first time, the timestamp of the currently detected video data frame is recorded as the first time of the target indicator light.

[0073] If the target detection model determines that the confidence of the second target video data frame is less than or equal to the preset confidence threshold, and the confidence of the previous video data frame of the second target video data frame is greater than the preset confidence threshold, the timestamp corresponding to the second target video data frame is determined to be the second time. That is, when the target detection model detects that the target indicator light goes out for the first time after being lit for the first time, the timestamp of the currently detected video data frame is recorded as the second time of the target indicator light.

[0074] It can be understood that when there are multiple detection targets, that is, when the target detection model needs to detect the indicator lights of multiple instruments, the target detection model can output a confidence level for each detection target, and record the first time and / or second time for the detection target that meets the recording timestamp.

[0075] In some embodiments, determining that the confidence of a target video data frame is greater than a preset confidence threshold may be by inputting a target video data frame into a target detection model, where the target video data frame is any frame of video data frame in a first object instance; and in response to detecting a target indicator light in a lit state in the target video data frame, determining that the confidence of the target video data frame is greater than a preset confidence threshold.

[0076] Determining that the confidence of the target video data frame is less than or equal to the preset confidence threshold may be: inputting the target video data frame into the target detection model, where the target video data frame is any frame of video data in the first object instance; in response to not detecting the target indicator light in the illuminated state in the target video data frame, determining that the confidence of the target video data frame is less than or equal to the preset confidence threshold. The value of the preset confidence threshold may be set according to actual needs, for example, set to 0.99.

[0077] In order to facilitate the test personnel to observe the instrument self-test process, the target indicator light detected by the target detection model can also be marked, and the marked image can be displayed in the test equipment.

[0078] Figure 4 This is another method flow chart of starting a target detection model to detect the state of a target indicator light in a smart cockpit provided by an embodiment of the present application. Figure 4 Steps S401 to S402 in the embodiment shown are Figure 2 Steps S201 to S202 in the illustrated embodiment are substantially the same and will not be described in detail herein. Figure 4 As shown, the method also includes the following steps:

[0079] In step S403, a first video stream signal is defined in the processing thread, and the first video stream signal is bound to a first custom signal slot.

[0080] In step S404, in response to determining that the confidence of the third target video data frame is greater than a preset confidence threshold, the target indicator light in the third target video data frame is marked to obtain a processed third target video data frame.

[0081] In step S405, the processed third target video data frame is transmitted to the first video stream signal.

[0082] In step S406, a display method is called to display the processed third target video data frame on the test device interface through the first custom signal slot.

[0083] In some embodiments of the present application, a first video stream signal may also be defined in a processing thread, for example, as handle_frame_signal, and the first video stream signal may be bound to a first custom signal slot. When the processing thread calls the target detection model to determine that the confidence of the third target video data frame in the first object instance is greater than a preset confidence threshold, the target indicator light in the third target video data frame may be marked to obtain a processed third target video data frame. The marking method may be selected according to actual needs, such as adding a regional frame line to the target indicator light, which is not limited here.

[0084] The processed third target video data frame is transmitted to the first video stream signal, and the test device calls the display method to display the processed third target video data frame on the test device interface through the first custom signal slot, so that the image marked with the target indicator light can be displayed in the test device.

[0085] On the other hand, when the processing thread calls the target detection model while transmitting the video frames to be displayed, there may be a lag problem. This is because the data frame needs to be inferred and detected by the target detection model before being sent back to the system for display.

[0086] To solve this problem, a video thread can be started in the test device, a second video stream signal can be defined in the video thread, and the second video stream signal can be bound to the second custom signal slot. After receiving the video stream start signal, the test device instantiates the second object to obtain the second object instance, and then saves the video data of the target interface to the second object instance. Furthermore, the video data frames in the second object instance can be sequentially transmitted to the second video stream signal.

[0087] At this time, the test device calls the display method to display the third target video data frame on the test device interface through the first custom signal slot and the second custom signal slot. The third target video data frame is the superimposed data frame of the original third target video data frame and the processed third target video data frame.

[0088] It is understandable that if there is no associated processed video data frame in the first custom signal slot, the test device only displays the original video data frame associated with the second custom signal slot. Meanwhile, the first target video data frame, the second target video data frame and the third target video data frame may be the same video data frame or different video data frames.

[0089] Figure 5 : is the overall flow chart of the self-check verification of the intelligent cockpit instrument provided in the embodiment of the present application. Figure 5As shown in the figure, the process includes first building a self-test verification system, starting the video stream when the instrument self-test verification is required, and starting AI detection, which calls the target detection model to detect the lighting status of the instrument indicator light. Next, the instrument starts self-testing, uses the AI ​​detection results to verify the self-test situation, and finally reports the verification results.

[0090] Figure 6 Schematic diagram of the structure of the self-checking and verification system provided by the embodiment of the present application. Figure 6 As shown, the self-test verification system includes external devices, clients, test equipment with a graphical interface, and model services. Among them, the external devices may include cameras and video stream acquisition applications, and the video stream acquisition device is a built-in screen recording application. The client includes a video stream display module, an AI detection echo module, an AI detection real-time log module, an abnormal prompt module, a visual report module, a video storage module, and an instrument log acquisition module. The model service module includes an instrument indicator light picture set acquisition module, a picture annotation module, and a YOLOv5 model service module. The external device and the client communicate with each other, and the test equipment with a graphical interface and the model service module communicate with each other.

[0091] In some implementations, the test device can use the QT graphical user interface application development interface framework to achieve video stream acquisition, target detection, and video stream display. The following takes the implementation of instrument self-test verification and test result display based on the QT framework as an example to describe the entire verification process in detail.

[0092] Figure 7 Schematic diagram of the workflow of the video thread provided in the embodiment of the present application. Figure 7 As shown, the video thread is used to process the original video stream, which includes the first parameter frame_signal and the second parameter cap, where frame_signal is a custom signal and cap is a thread parameter, which refers to a camera or video stream object. In the video thread, you can first define the signal frame_signal used to transmit each frame of the video stream, and then initialize the cap object. Next, loop read the cap object information and return each frame data frame. Finally, pass the frame to the signal frame_signal.

[0093] That is to say, in CameraThread, you can first define the signal frame_signal for transmitting each frame of the video stream, and bind each frame of data in order to communicate with the video stream object. Then initialize the cap object and define the cap video stream object. At this time, no instance is involved. The instance is performed in the self-check verification system display process. Next, loop read the cap object information and return each frame of data frame, so as to loop to obtain each frame of the video stream data, and finally pass the frame to the signal frame_signal, that is, pass each frame of data to the custom signal slot.

[0094] Figure 8 Schematic diagram of the process flow of the processing thread provided in the embodiment of the present application. Figure 8 As shown, the processing thread is used to process the instrument self-test verification, which includes the first parameter handle_frame_signal and the second parameter cap, where handle_frame_signal is a custom signal and cap is a thread parameter, which refers to the camera or video stream object. In the processing thread, the signal handle_frame_signal used to transmit each frame of the video stream can be defined, and then the cap object is initialized. Next, the cap object information is read in a loop and each frame of data frame is returned. Each frame of data frame is sent to the YOLOv5 model for inference, and the target appearance area is drawn on the frame after the model recognizes the target. Finally, the frame with the target appearance area drawn is passed to handle_frame_signal.

[0095] That is to say, in HandleThread, you can first define the signal handle_frame_signal used to transmit each frame of the video stream, and bind each frame of data in order to communicate with the video stream object. Then initialize the cap object, that is, define the cap video stream object. No instance is involved at this time, and the instance is performed in the self-test verification system display process. Next, read the cap object information in a loop and return the data frame for each frame, that is, loop to obtain the data of each frame of the video stream. Then use Yolov5 to perform model inference, and draw the target appearance area on the frame after identifying the target. In an example, opencv can be used to detect the target image frame, so that the system can echo the AI ​​detection situation. Finally, pass the frame to handle_frame_signal, that is, pass the video frame to the custom signal slot.

[0096] Fig. 9 Schematic diagram of the workflow of the self-checking verification display method provided in the embodiment of the present application. Fig. 9As shown, the self-check verification display method may include a parameter show_frame. The method uses opencv to process the picture frame. Once the picture frame frame is passed in, it is captured and displayed on the client (ie, the test device) interface.

[0097] In the self-test verification display method, the CameraThread thread instance can be used to bind the original video stream thread to the system UI display screen CameraThread.frame_signal.connect(show_frame) through a custom signal slot, so that the original video stream is displayed on the test device interface. On the other hand, the HandleThread thread instance can also be used to bind the video stream processing thread to the system UI display screen HandleThread.handle_frame_signal.connect(show_frame) through a custom signal slot, so that the target area is drawn on the test device interface after the AI ​​detects the target.

[0098] That is to say, in the show_frame method, the cap object can be passed into the CameraThread through the CameraThread thread instance. At this time, the original video stream thread binds the system UI display screen through the custom signal slot. At the same time, the cap object can also be passed into the CameraThread through the HandleThread thread instance. At this time, the AI ​​detection and processing video stream thread binds the system UI display screen through the custom signal slot. The two signals can be superimposed and displayed. When there is no display screen after AI processing, the original screen is displayed. When there is a display screen after AI processing, since the AI ​​processing is to add an identifier to the original image frame, superimposing a signal of the original screen will not affect the display of the processed screen.

[0099] Fig.10 Schematic diagram of the process flow of AI detection processing provided by the embodiment of the present application. Fig.10 As shown, if the AI ​​model is a YOLOv5 model, you can first start the YOLOv5 model and input the picture stream frame into the model. Perform inference recognition on the input picture stream frame. If the confidence of the picture frame detected in the picture stream is greater than 0.99, the current timestamp is recorded as the start time from the first frame being detected, and the subsequent detection of the target does not participate in the timing. Otherwise, the current timestamp is recorded as the end time of the detection from the last frame that did not detect the target, and the subsequent detection does not participate in the timing. Then the difference between the detection end time and the detection start time is used as the continuous display time obtained by the detection, and the detection process is exited.

[0100] In other words, the frame method of YOLOv5 inference in the video stream can infer one target or multiple targets. If the confidence of the inference of the picture frame is greater than 0.99, the current timestamp is recorded as the start time from the first frame detection, and the subsequent detection of the target does not participate in the timing until the continuous detection does not meet the confidence greater than 0.99 to obtain the end time. The difference between the detection end time and the detection start time is used as the continuous display time obtained by the detection.

[0101] Fig.11 1 is a flow chart of implementing instrument self-check verification and test result display based on QT framework provided in the embodiment of the present application. Fig.11 As shown, you can first open the video stream, for example, use the interface button control to open the video stream. Then instantiate the cap object, for example, open the video stream object for instance processing, such as: camera, cv2.VideoCapture. Next, start the CameraThread thread, implement the picture frame frame data transmission, and pass the frame_signal to the picture frame frame, where the picture frame of the original video stream can be transmitted through the custom signal slot.

[0102] Then you can start AI detection, for example, start Yolov5 model detection. After starting Yolov5 model detection, you can start HandleThread thread, and pass the processed image frame frame to handle_frame_signal, where the processed video stream image frame frame can be passed through the custom signal slot. Finally, the real-time detection screen of AI detection is presented in the self-test verification system of the test equipment, and a report is generated.

[0103] Among them, the self-check verification system can obtain each frame in the original video stream and directly bind it to the self-check system display method (show_frame), and the CameraThread thread method has a cap object parameter. When the "Start Video Stream" button is triggered on the self-check verification system interface, the specific camera or video stream is instantiated, and the instantiated cap object is passed to the CameraThread thread method. At this time, CameraThread is turned on, and the self-check system displays the camera device or video stream screen.

[0104] At the same time, the self-check verification system can also process the image frames connected to the YOLOv5 model inference. After the processing is completed, each frame is obtained and directly bound to the self-check system display method (show_frame). The HandleThread thread method has a cap object parameter. After "turning on the video stream" in the self-check verification system interface, "turning on AI detection" is then done, and the cap object after the instance is passed to the HandleThread thread method. At this time, HandleThread is turned on, and the self-check system displays the detection effect on the interface after the Yolov5 model inference detects the target. The effect display can be, for example, an interface prompt, a target drawing frame, etc.

[0105] Furthermore, the self-check verification system passes the picture frame to the show_frame(frame) method through the CameraThread and HandleThread thread methods through custom signal slot binding, thereby solving the jamming problem.

[0106] Fig.12 Detailed flow chart of the self-check verification of the intelligent cockpit instrument provided in the embodiment of the present application. Fig.12 As shown, when performing instrument self-test verification, you can first turn on AI detection, such as using the YOLOv5 model, and then monitor the instrument screen in real time to capture the instrument self-test process.

[0107] In some embodiments, a program can be used in the test equipment to control the closure of the vehicle power line, such as controlling a relay-like device through the device COM port (Communication Port, serial communication port), simulating the vehicle startup, thereby simulating the instrument startup process. Next, start the system client detection, and continuously detect the continuous display time of the indicator light by detecting the instrument self-test screen. Determine whether the continuous display time is equal to the expected result, that is, compare the continuous display time of the instrument self-test indicator light detected by the system with the expected time. If it meets the requirements, the verification is passed and a report is generated; if it does not meet the requirements, obtain the vehicle log and video for reference by the instrument developer and generate a report.

[0108] In some embodiments of the present application, if the test device does not receive at least one of the first time and the second time of the target indicator light, or determines that the absolute value of the difference between the detection time of the target indicator light and the preset display time is less than a preset threshold, it can be determined that the instrument self-test corresponding to the target indicator light has failed.

[0109] In some embodiments, the target detection model may be a trained target detection model. The trained detection model is trained in the following manner: obtaining training samples, the training samples at least including images of self-test indicator lights of different instruments in different types of smart cockpits when they are lit; obtaining a pre-trained target detection model; and using the training samples to train the target detection model until the target detection model has a recognition accuracy greater than a preset recognition threshold for any indicator light in a lit state.

[0110] Table 2 shows some examples of training samples:

[0111] Table 2 Some examples of training samples

[0112]

[0113]

[0114] In one example, during training, 70% of the data in the sample can be used as the training set, 20% of the data as the validation set, and 10% of the data as the test set. During training, comprehensive evaluation is performed on the reference precision and recall curves, average precision, confusion matrix, etc. to determine whether the model has converged.

[0115] By adopting the technical solution of the embodiment of the present application, the display duration of the instrument self-test indicator light is accurately detected by using the target detection model, which solves the misdetection caused by subjective factors in the instrument self-test verification process, realizes the full automation of the instrument self-test indicator light verification process, and solves the serious problems of low efficiency and high labor costs. At the same time, the instrument self-test indicator light verification method can be reused in various vehicle models, and the full set of automated instrument self-test verification process solutions improves the implementation efficiency and solves the problem of poor versatility.

[0116] Furthermore, the technical solution of the embodiment of the present application adopts an AI model combined with a program to control the startup of the instrument, which can obtain the continuous display time of the indicator light during the instrument self-test process, reducing the accuracy problem of manual verification. A set of program automation control combined with AI technology instrument self-test process verification method has been formed. The operation is simple. You only need to operate the self-test verification system / program to automatically verify the instrument self-test process and generate a report, which greatly reduces human resource costs. By controlling the startup of the vehicle computer through the program, the instrument screen is indirectly presented, and the AI ​​technology is run in advance to train the target indicator light, which can be quickly implemented in the instrument self-test verification link, improve the verification efficiency, and the trained indicator light content can be used in the instrument self-test verification process of each vehicle model. Automatically generate a customized verification report and record the video of the entire verification process in real time.

[0117] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0118] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0119] Fig.13 Schematic diagram of an intelligent cockpit instrument self-checking device provided in an embodiment of the present application. Fig.13 As shown, the device comprises:

[0120] The startup module 1301 is configured to start the target detection model to detect the status of the target indicator light in the smart cockpit in response to determining that the smart cockpit starts the instrument self-check. The target indicator light is the self-check indicator light of any N instruments in the smart cockpit, where N is a positive integer.

[0121] The receiving module 1302 is configured to receive a first time and a second time returned by the target detection model, wherein the first time is the time when the target indicator light is first lit, and the second time is the time when the target indicator light is first extinguished after being lit.

[0122] The determination module 1303 is configured to determine a time difference between the second time and the first time as a detection duration of the target indicator light.

[0123] The detection module 1304 is configured to determine that the instrument self-test corresponding to the target indicator light has passed in response to determining that the absolute value of the difference between the detection duration and the preset display duration is less than a preset threshold.

[0124] According to the technical solution provided in the embodiment of the present application, the target detection model is started to detect the status of the target indicator light in the smart cockpit under the condition that the instrument self-test of the smart cockpit is turned on, and the first time when the target indicator light is first lit and the second time when it is first extinguished after lighting are obtained. When the absolute value of the time difference between the second time and the first time is less than a preset threshold, it is determined that the instrument self-test corresponding to the target indicator light has passed, thereby realizing smart cockpit instrument self-test verification based on interface detection, improving the accuracy of self-test result judgment, and improving judgment efficiency.

[0125] In some embodiments, a target detection model is started to detect the status of a target indicator light in a smart cockpit, including: sending a startup message to a video recording application so that the video recording application records the video of a target interface in real time, where the target interface includes at least a target indicator light; starting a processing thread, obtaining video data of the target interface in the processing thread, calling a target detection model to detect the video data of the target interface, and obtaining the status of the target indicator light.

[0126] In some embodiments, a target detection model is called in a processing thread to detect video data of a target interface to obtain a state of a target indicator light, including: writing the video data of the target interface to a first object of the processing thread; instantiating the first object in response to receiving a target detection start signal; calling the target detection model to detect the confidence of each video data frame in the first object instance frame by frame; in response to determining that the confidence of the first target video data frame is greater than a preset confidence threshold, and the confidences of the preceding video data frames of the first target video data frame are less than or equal to the preset confidence threshold, determining that the timestamp corresponding to the first target video data frame is the first time; in response to determining that the confidence of the second target video data frame is less than or equal to the preset confidence threshold, and the confidence of the previous video data frame of the second target video data frame is greater than the preset confidence threshold, determining that the timestamp corresponding to the second target video data frame is the second time.

[0127] In some embodiments, determining that the confidence of a target video data frame is greater than a preset confidence threshold comprises: inputting a target video data frame into a target detection model, the target video data frame being any one video data frame in a first object instance; in response to detecting a target indicator light in a lit state in the target video data frame, determining that the confidence of the target video data frame is greater than a preset confidence threshold; determining that the confidence of the target video data frame is less than or equal to a preset confidence threshold comprises: inputting a target video data frame into a target detection model, the target video data frame being any one video data frame in a first object instance; in response to not detecting a target indicator light in a lit state in the target video data frame, determining that the confidence of the target video data frame is less than or equal to a preset confidence threshold.

[0128] In some implementations, it also includes: defining a first video stream signal in a processing thread, and binding the first video stream signal to a first custom signal slot; in response to determining that the confidence of a third target video data frame is greater than a preset confidence threshold, marking a target indicator light in the third target video data frame to obtain a processed third target video data frame; passing the processed third target video data frame into the first video stream signal; and calling a display method to display the processed third target video data frame on a test device interface through the first custom signal slot.

[0129] In some implementations, it also includes: starting a video thread, defining a second video stream signal in the video thread, and binding the second video stream signal to a second custom signal slot; in response to receiving the video stream start signal, instantiating a second object, and saving the video data of the target interface to the second object instance; passing the video data frames in the second object instance into the second video stream signal in sequence; calling a display method, displaying a third target video data frame on the test device interface through the first custom signal slot and the second custom signal slot, the third target video data frame being a superimposed data frame of the original third target video data frame and the processed third target video data frame.

[0130] In some implementations, the method further includes: in response to not receiving at least one of the first time and the second time, or in response to determining that the absolute value of the difference between the detection time and the preset display time is less than a preset threshold, determining that the instrument self-test corresponding to the target indicator light has failed.

[0131] In some embodiments, the target detection model is a trained target detection model; the trained detection model is trained in the following manner: obtaining training samples, the training samples at least including images of self-test indicator lights of different instruments in different types of smart cockpits when they are lit; obtaining a pre-trained target detection model; using the training samples to train the target detection model until the target detection model's recognition accuracy for any indicator light in a lit state is greater than a preset recognition threshold.

[0132] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0133] Fig.14 Schematic diagram of an electronic device provided in an embodiment of the present application. Fig.14 As shown, the electronic device 14 of this embodiment includes: a processor 1401, a memory 1402, and a computer program 1403 stored in the memory 1402 and executable on the processor 1401. When the processor 1401 executes the computer program 1403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 1401 executes the computer program 1403, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0134] The electronic device 14 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 14 may include, but is not limited to, a processor 1401 and a memory 1402. Those skilled in the art will appreciate that Fig.14 The electronic device 14 is merely an example and does not limit the electronic device 14 , and may include more or less components than those shown in the figure, or different components.

[0135] Processor 1401 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0136] The memory 1402 may be an internal storage unit of the electronic device 14, for example, a hard disk or memory of the electronic device 14. The memory 1402 may also be an external storage device of the electronic device 14, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the electronic device 14. The memory 1402 may also include both an internal storage unit of the electronic device 14 and an external storage device. The memory 1402 is used to store computer programs and other programs and data required by the electronic device.

[0137] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0139] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for self-checking an intelligent cockpit instrument, characterized in that: The method is performed by a test device and includes: In response to determining that the smart cockpit starts the instrument self-check, starting the target detection model to detect the state of the target indicator light in the smart cockpit, the target indicator light is the self-check indicator light of any N instruments in the smart cockpit, where N is a positive integer; Receive a first time and a second time returned by the target detection model, wherein the first time is the time when the target indicator light is first lit, and the second time is the time when the target indicator light is first extinguished after being lit; Determine a time difference between the second time and the first time as a detection duration of the target indicator light; In response to determining that the absolute value of the difference between the detection time and the preset display time is less than a preset threshold, it is determined that the instrument self-test corresponding to the target indicator light passes.

2. The method according to claim 1, characterized in that The starting target detection model detects the state of the target indicator light in the smart cockpit, including: Sending a start message to a video recording application so that the video recording application records a video of a target interface in real time, wherein the target interface at least includes the target indicator light; A processing thread is started, video data of the target interface is obtained in the processing thread, the target detection model is called to detect the video data of the target interface, and the state of the target indicator light is obtained.

3. The method according to claim 2, characterized in that Calling the target detection model in the processing thread to detect the video data of the target interface and obtaining the state of the target indicator light includes: Writing the video data of the target interface into the first object of the processing thread; In response to receiving an on-target detection signal, instantiating the first object; Calling the target detection model to detect the confidence of each video data frame in the first object instance frame by frame; In response to determining that the confidence of the first target video data frame is greater than a preset confidence threshold, and the confidences of the preceding video data frames of the first target video data frame are all less than or equal to the preset confidence threshold, determining that the timestamp corresponding to the first target video data frame is the first time; In response to determining that the confidence of the second target video data frame is less than or equal to the preset confidence threshold, and the confidence of the previous video data frame of the second target video data frame is greater than the preset confidence threshold, the timestamp corresponding to the second target video data frame is determined to be the second time.

4. The method according to claim 3, characterized in that Determining that the confidence of the target video data frame is greater than a preset confidence threshold includes: Inputting a target video data frame into the target detection model, wherein the target video data frame is any frame of video data in the first object instance; In response to detecting a target indicator light in a lit state in the target video data frame, determining that the confidence level of the target video data frame is greater than a preset confidence threshold; or Determining whether the confidence level of the target video data frame is less than or equal to a preset confidence level threshold includes: Inputting a target video data frame into the target detection model, wherein the target video data frame is any frame of video data in the first object instance; In response to not detecting a target indicator light in a lit state in the target video data frame, it is determined that the confidence of the target video data frame is less than or equal to a preset confidence threshold.

5. The method according to claim 2, characterized in that: The method further comprises: Defining a first video stream signal in the processing thread, and binding the first video stream signal to a first custom signal slot; In response to determining that the confidence of the third target video data frame is greater than a preset confidence threshold, marking the target indicator light in the third target video data frame to obtain a processed third target video data frame; Transmitting the processed third target video data frame into the first video stream signal; A display method is called to display the processed third target video data frame on the test device interface through the first custom signal slot.

6. The method according to claim 5, characterized in that The method further comprises: Start a video thread, define a second video stream signal in the video thread, and bind the second video stream signal to a second custom signal slot; In response to receiving a signal to start video streaming, instantiate the second object, and save the video data of the target interface to the second object instance; sequentially transferring the video data frames in the second object instance into the second video stream signal; Calling a display method, displaying a third target video data frame on the test device interface through the first custom signal slot and the second custom signal slot, wherein the third target video data frame is a superimposed data frame of the original third target video data frame and the processed third target video data frame.

7. The method according to claim 1, characterized in that The method further comprises: In response to not receiving at least one of the first time and the second time, or in response to determining that the absolute value of the difference between the detection duration and the preset display duration is less than a preset threshold, it is determined that the instrument self-test corresponding to the target indicator light has failed.

8. The method according to any one of claims 1 to 7, characterized in that The target detection model is a trained target detection model; The trained detection model is trained in the following way: Acquire training samples, where the training samples at least include images of self-test indicator lights of different instruments in different types of smart cockpits when they are lit; Get the pre-trained object detection model; The target detection model is trained using the training samples until the recognition accuracy of the target detection model for any indicator light in a lit state is greater than a preset recognition threshold.

9. An intelligent cockpit instrument self-checking device, characterized in that: include: a start-up module configured to start a target detection model to detect a state of a target indicator light in the smart cockpit in response to determining that the smart cockpit starts an instrument self-check, wherein the target indicator light is a self-check indicator light of any N instruments in the smart cockpit, where N is a positive integer; A receiving module is configured to receive a first time and a second time returned by the target detection model, wherein the first time is the time when the target indicator light is first lit, and the second time is the time when the target indicator light is first extinguished after being lit; A determination module is configured to determine a time difference between the second time and the first time as a detection duration of the target indicator light; The detection module is configured to determine that the instrument self-test corresponding to the target indicator light has passed in response to determining that the absolute value of the difference between the detection duration and the preset display duration is less than a preset threshold.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images