Visual model reasoning effect real-time verification method and related product

Abstract

The invention discloses a visual model reasoning effect real-time verification method and related products, which can be applied to the technical field of data processing, and the method comprises the following steps: obtaining video stream data of a target image; processing the original image data of the target image based on a visual perception function module to obtain first structured data; performing supplementary processing on the first structured data to obtain second structured data; based on a transmission layer UDP protocol, the video stream data, the first structured data and the second structured data are sent to a computer end in real time in a serialization operation mode, visualization processing is carried out, the video stream data, the first structured data and the second structured data after visualization processing are matched according to timestamps, and the video stream data, the first structured data and the second structured data are obtained. And obtaining visual verification data and displaying the visual verification data. In this way, real-time visualization of the visual perception function module from the vehicle controller to the computer end is achieved based on the UDP protocol, and the verification efficiency of the visual model reasoning effect is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for real-time verification of visual model reasoning effects and related products. Background Art

[0002] In the development of intelligent driving assistance systems, the visual perception function module is deployed on the actual vehicle controller, receives image data from several cameras, and completes the corresponding perception tasks.

[0003] In the existing real-time verification method of visual model reasoning effect, the visual perception function module ultimately runs on the vehicle controller, and it is impossible to connect an external display to directly observe the real-time effect of the visual perception function module, which results in the visual algorithm development engineers being unable to view the reasoning effect of the visual perception function module in a timely manner, reducing the verification efficiency of the visual model reasoning effect. However, during the development process, the visual algorithm development engineers need to observe the effect of the visual perception function module on the actual vehicle to discover the deficiencies in the module, so as to propose reasonable solutions for iterative optimization.

[0004] Therefore, how to improve the verification efficiency of visual model reasoning effects is an issue that technical personnel in this field urgently need to solve. Summary of the invention

[0005] Based on the above problems, the present application provides a method and related products for real-time verification of visual model reasoning effects, which realizes real-time visualization of the visual perception function module from the vehicle controller to the computer based on the UDP protocol, thereby solving the problem of low verification efficiency of the visual model reasoning effects in the prior art.

[0006] In a first aspect, the present application provides a method for real-time verification of visual model reasoning effects, comprising:

[0007] Get the video stream data of the target image;

[0008] Processing the original image data of the target image based on the visual perception function module to obtain first structured data;

[0009] Performing supplementary processing on the first structured data to obtain second structured data;

[0010] Based on the transport layer UDP protocol, the video stream data, the first structured data and the second structured data are sent to the computer in real time by serialization operation;

[0011] Based on the computer, respectively, the video stream data, the first structured data, and the second structured data are visualized, and the visualized video stream data, the first structured data, and the second structured data are matched according to timestamps to obtain visualization verification data;

[0012] Based on the visual interface of the computer, the visual verification data is displayed.

[0013] Optionally, before obtaining the video stream data of the target image, the method further includes:

[0014] Obtaining the image address of the target image based on inter-core communication;

[0015] Based on the image address, the complete image data of the target image is acquired in an image format to achieve acquisition of the target image.

[0016] Optionally, the acquiring of the video stream data of the target image includes:

[0017] The complete image data is compressed to obtain a video stream in a predetermined format corresponding to the complete image data, thereby acquiring the video stream data of the target image.

[0018] Optionally, the processing of the original image data of the target image based on the visual perception function module to obtain the first structured data includes:

[0019] Preprocessing the original image data of the target image based on the visual perception function module to obtain first image data that meets the input of the visual perception function model;

[0020] The first image data is inferred using the visual perception function model, and first structured data is obtained through tensor processing.

[0021] Optionally, based on the transport layer UDP protocol, the video stream data, the first structured data, and the second structured data are sent to the computer in real time by serialization operation, including:

[0022] Based on the transport layer UDP protocol, the first structured data is sent to the computer in real time using the first channel;

[0023] Based on the UDP protocol, the second structured data is sent to the computer terminal in real time using the second channel;

[0024] Based on the UDP protocol, the video stream data is sent to the computer terminal in real time using a third channel.

[0025] Optionally, the method further includes:

[0026] Based on the visual interface of the computer, in response to a start recording instruction, at least one of the video stream data, the first structured data, and the second structured data is recorded to obtain recorded data;

[0027] Based on the visual interface of the computer, the recorded data is saved locally in response to a stop recording instruction.

[0028] Optionally, the method further includes:

[0029] Based on the visual interface of the computer, the recorded data is parsed in response to the parsing instruction to obtain parsed image data;

[0030] Based on the visualization interface of the computer, the visualization effect is exported in a video format in response to an export instruction.

[0031] In a second aspect, the present application provides a device for real-time verification of visual model reasoning effect, characterized in that it includes:

[0032] An image encoding module, used for acquiring video stream data of a target image;

[0033] A first processing module, used for processing the original image data of the target image based on the visual perception function module to obtain first structured data;

[0034] A second processing module, configured to perform supplementary processing on the first structured data to obtain second structured data;

[0035] A data publishing module, used for sending the video stream data, the first structured data and the second structured data to the computer terminal in real time by serialization operation based on the transport layer UDP protocol;

[0036] A visualization processing module, for performing visualization processing on the video stream data, the first structured data and the second structured data respectively based on a computer, and matching the visualized video stream data, the first structured data and the second structured data according to timestamps to obtain visualization verification data;

[0037] The display module is used to display the visual verification data based on the visual interface of the computer.

[0038] In a third aspect, the present application provides a device for real-time verification of visual model reasoning effect, characterized in that it includes:

[0039] Memory for storing computer programs;

[0040] A processor is used to implement the steps of the method for real-time verification of visual model reasoning effect as described in any of the above items when executing the computer program.

[0041] In a fourth aspect, the present application provides a readable storage medium, characterized in that a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of the method for real-time verification of visual model reasoning effects as described in any of the above items are implemented.

[0042] It can be seen from the above technical solutions that compared with the prior art, the present application has the following advantages:

[0043] This application can effectively address the problem that the visual perception function module cannot directly display the effect on the vehicle controller, and proposes a real-time verification method for the reasoning effect of the visual perception function module model to achieve visualization from the vehicle controller to the computer. Specifically, a visualization image encoding module independent of the visual perception function module is added to avoid additional expenses on the frame rate and load of the visual perception function module. At the same time, multiple visual perception function modules can obtain images for visualization based on this module. Through the four-camera solution, the visualization frame rate reaches 15+FPS while ensuring the visualization clarity, ensuring the real-time display of the visual perception function module. It supports real-vehicle log recording, offline playback on the computer, and video export, which is convenient for visual algorithm engineers to review and effectively reduce the labor cost of testing. It supports image analysis, which is convenient for visual algorithm engineers to view the model reasoning effect on the computer and compare it with the model reasoning on the vehicle controller. It supports the data re-injection of the visual perception function module in the vehicle controller, which is convenient for visual algorithm engineers to verify the iteratively optimized model or algorithm without real-vehicle testing again. Finally, it can facilitate visual algorithm engineers to effectively troubleshoot and analyze problems in each module, so that each module can solve the problem in a targeted manner, playing an important role in the effective closed loop of "discovering problems-analyzing and troubleshooting-proposing solutions-real-vehicle verification". BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a method for real-time verification of visual model reasoning effect provided in this application;

[0045] Figure 2 A schematic diagram of a method for acquiring video stream data of a target image provided by the present application;

[0046] Figure 3 A schematic diagram of a visual perception function module provided in the present application processing raw image data of a target image;

[0047] Figure 4 A schematic diagram of data transmission provided for this application;

[0048] Figure 5 A schematic diagram of a visualization process provided for this application;

[0049] Figure 6 A schematic diagram of a visual interface provided for this application;

[0050] Figure 7 A schematic diagram of a data re-injection provided for this application;

[0051] Figure 8 A schematic diagram of the structure of a device for real-time verification of visual model reasoning effects provided in this application. DETAILED DESCRIPTION

[0052] As mentioned above, the existing real-time verification method for visual model reasoning effect has low verification efficiency. Specifically, in the existing real-time verification method for visual model reasoning effect, the visual perception function module ultimately runs on the vehicle controller, and an external display cannot be connected to directly observe the real-time effect of the visual perception function module, which results in the visual algorithm development engineer being unable to view the reasoning effect of the visual perception function module in a timely manner, reducing the verification efficiency of the visual model reasoning effect. However, during the development process, the visual algorithm development engineer needs to observe the effect of the visual perception function module on the actual vehicle to discover the deficiencies in the module, so as to propose reasonable solutions for iterative optimization.

[0053] To solve the above problems, the present application provides a method for real-time verification of the reasoning effect of a visual model, including: first obtaining the video stream data of the target image, and processing the original image data of the target image based on the visual perception function module to obtain the first structured data. Then the first structured data is supplemented to obtain the second structured data. Based on the transport layer UDP protocol, the video stream data, the first structured data and the second structured data are sent to the computer in real time through serialization operations for visualization. Finally, the video stream data, the first structured data and the second structured data after visualization are matched according to the timestamp to obtain the visualization verification data and display it.

[0054] In this way, real-time visualization of the visual perception function module from the vehicle controller to the computer is realized based on the UDP protocol, which improves the verification efficiency of the visual model reasoning effect.

[0055] It should be noted that the method and related products provided by this application for real-time verification of visual model reasoning effects can be applied to the field of data processing technology. The above is only an example and does not limit the application field of the method and related products provided by this application for real-time verification of visual model reasoning effects.

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] Figure 1 A flowchart of a method for real-time verification of visual model reasoning effect provided in this application. Figure 1 As shown, a method for real-time verification of visual model reasoning effect provided by the present application may include:

[0058] S101: Acquire video stream data of a target image.

[0059] In practical applications, the visual perception function module belongs to deep learning technology, which is data-driven. For vision, this data is an image. Therefore, the visualization of the visual model reasoning effect is inseparable from the image. The image will be passed to the visual perception function module as input, but the additional image processing and publishing will have a greater impact on the frame rate and load of the visual perception function module, resulting in inaccurate performance testing. To avoid such problems, we designed a visualization image encoding module independent of the visual perception function module in the device for real-time verification of the visual model reasoning effect. This design also allows multiple visual perception function modules to obtain images based on this module for visualization. Therefore, a device for real-time verification of the visual model reasoning effect provided in the present application first needs to obtain the video stream data of the target image. Among them, the target image is the image required for the visual perception function module to make predictions.

[0060] In addition, since the methods for acquiring the target image are not the same, the present application may describe one possible acquisition method.

[0061] In one case, regarding how to obtain a target image, the method further includes:

[0062] Obtaining the image address of the target image based on inter-core communication;

[0063] Based on the image address, the complete image data of the target image is acquired in an image format to achieve acquisition of the target image.

[0064] In practical applications, a device for real-time verification of visual model reasoning effects can obtain the image address of a target image based on inter-core communication, and based on the image address, obtain the complete image data of the target image in an image format, thereby achieving the acquisition of the target image.

[0065] In addition, since there are different ways to obtain video stream data, this application can describe a possible acquisition method.

[0066] In one case, with respect to how to obtain video stream data, correspondingly, S101: obtaining video stream data of a target image, specifically includes:

[0067] The complete image data is compressed to obtain a video stream in a predetermined format corresponding to the complete image data, thereby acquiring the video stream data of the target image.

[0068] In practical applications, Figure 2 A schematic diagram of a method for acquiring video stream data of a target image provided by the present application. Figure 2 As shown, the camera on the actual vehicle captures images and stores them in a preset storage address. When the device for real-time verification of the visual model reasoning effect obtains the video stream data of the target image, it is necessary to first obtain the complete data of the image from the storage address of the image through inter-core communication, that is, to obtain the complete image data. It should be noted that the image is the communication data that occupies the largest space in the published message. Taking into account the real-time performance, bandwidth limitations and the possibility of packet loss, this application compresses the complete image data to obtain the video stream data. Specifically, the real-time verification device for the visual model reasoning effect compresses the complete image data, and the compression ratio can generally reach 30+. Finally, it compresses the video stream in a predetermined format corresponding to the complete image data, thereby obtaining the video stream data of the target image. It should be noted that the predetermined format is generally the H.264 format.

[0069] S102: Processing the original image data of the target image based on the visual perception function module to obtain first structured data.

[0070] In practical applications, combined with the above-mentioned method for acquiring images, the original image data of the target image is first acquired based on the visual perception function module, and then the visual perception function module performs real-time processing based on the original image data. Specifically, the device for real-time verification of the visual model reasoning effect performs a series of data processing on the original image data based on the visual perception function module to obtain structured data for reflecting the reasoning effect.

[0071] In addition, since the methods for processing the original image data of the target image are not the same, the present application can describe a possible processing method.

[0072] In one case, with respect to how to process the original image data of the target image, correspondingly, S102: processing the original image data of the target image based on the visual perception function module to obtain first structured data specifically includes:

[0073] Preprocessing the original image data of the target image based on the visual perception function module to obtain first image data that meets the input of the visual perception function model;

[0074] The first image data is inferred using the visual perception function model, and first structured data is obtained through tensor processing.

[0075] In practical applications, visual perception function modules (such as target detection, semantic segmentation, scene classification, etc.) usually include the following operations: first, the image is preprocessed to obtain an image that meets the model input, then the deployed model is called to reason about the image, and finally tensor processing is performed to complete the corresponding task. The input of this module is image data, and the output is structured data. Different visual perception function modules contain different information in the structured data. Taking target detection as an example, the output structured data mainly includes 3D information of the target object, target category, etc. Figure 3 A schematic diagram of a visual perception function module provided in this application processing raw image data of a target image. Figure 3 As shown, the device for real-time verification of visual model reasoning effect provided by the present application first pre-processes the original image data of the target image based on the visual perception function module to obtain image data that meets the input of the visual perception function model, that is, obtains the first image data. Then the visual perception function model performs model reasoning on the first image data to obtain a tensor, and finally processes it to obtain structured data, that is, obtains the first structured data.

[0076] S103: Perform supplementary processing on the first structured data to obtain second structured data.

[0077] In practical applications, the device for real-time verification of the reasoning effect of the visual model can supplement the visual perception function model. Specifically, the visual perception function model mainly takes a single frame image as input, which belongs to the spatial dimension, but does not consider the correlation between the previous and next frame images on the time axis. Therefore, the device for real-time verification of the reasoning effect of the visual model provided in this application has a post-processing function, which can supplement the first structured data obtained by processing based on the visual perception function module to obtain the second structured data. Taking target detection as an example, post-processing mainly realizes the process of multi-target tracking. That is, the first structured data output by the visual perception function module is output as a more stable and effective second structured data after post-processing operations.

[0078] S104: Based on the transport layer UDP protocol, the video stream data, the first structured data and the second structured data are sent to the computer in real time by serialization operation.

[0079] In practical applications, the device for real-time verification of the reasoning effect of the visual model needs to establish a connection between the vehicle controller and the computer in order to realize the real-time visualization of the reasoning effect. Specifically, based on the transport layer UDP protocol communication technology, the video stream data used to characterize the reasoning effect, the first structured data, and the second structured data are sent to the computer in real time through serialization operations.

[0080] In addition, since the methods of sending to the computer are different, this application can explain one possible sending method.

[0081] In one case, regarding how to send data to a computer, S104: based on the transport layer UDP protocol, the video stream data, the first structured data, and the second structured data are sent to the computer in real time by serialization operation, specifically including:

[0082] Based on the transport layer UDP protocol, the first structured data is sent to the computer in real time using the first channel;

[0083] Based on the UDP protocol, the second structured data is sent to the computer terminal in real time using the second channel;

[0084] Based on the UDP protocol, the video stream data is sent to the computer terminal in real time using a third channel.

[0085] In practical applications, Figure 4 A schematic diagram of data transmission provided by this application. Figure 4 As shown, the present application publishes data through three channels based on UDP protocol communication technology. Specifically, the device for real-time verification of visual model reasoning effect is based on UDP protocol communication technology, using the first channel to send the first structured data to the computer end in real time, using the second channel to send the second structured data to the computer end in real time, and using the third channel to send the video stream data to the computer end in real time.

[0086] S105: Based on the computer, the video stream data, the first structured data and the second structured data are respectively visualized, and the visualized video stream data, the first structured data and the second structured data are matched according to timestamps to obtain visualization verification data.

[0087] In actual applications, after the data is sent to the computer, the device for real-time verification of the visual model reasoning effect needs to perform visualization processing based on the transmitted data, and then match the visualized video stream data, the first structured data and the second structured data according to the timestamp to obtain visualization verification data. Figure 5 A schematic diagram of a visualization process provided by this application. Figure 5 As shown, the transmission data is deserialized on the computer side, so that the computer side receives the video stream data, the first structured data and the second structured data. In order to avoid the situation of packet loss of the image video stream, an independent thread is used to subscribe to the video stream data transmitted in the third channel. The video stream data is decoded, the image format is converted, and the image data is dedistorted using the XML calibration file. Then, in order to ensure that there is no deviation in the visualization effect, the dedistorted image needs to be scaled to keep the same size as the model input image. For the first structured data, the 8 corner points are transformed to obtain a bird's-eye view, the coordinate system is transformed to obtain the 3D frame of the target object on the image, and the image 2D frame and category are obtained and cached. For the second structured data, the 8 corner points are transformed to obtain a bird's-eye view and the coordinate system is transformed to obtain the 3D frame of the target object on the image and cached. After completing the above operations, all processed data are matched according to the timestamp to obtain visual verification data, and then the button on the visualization interface is linked to respond to the user's button click, and the corresponding visual verification data is displayed accordingly. It should be noted that since there will be a certain delay in subscribing to the video stream data compared to the first structured data and the second structured data on the timeline, a cache is set for the structured data. The cache size is determined based on the actual application and is used to match the image data with the structured data. The matching strategy is that the image has its own timestamp. When the timestamps of the three-channel data are the same, it indicates that the match is successful and can be visualized.

[0088] In addition, since visualization interfaces are not all the same, the present application may describe a possible visualization interface.

[0089] Figure 6 A schematic diagram of a visual interface provided by this application. Figure 6 As shown, the visualization interface has multiple buttons and corresponding display areas. The buttons include specific test time, frame rate and target remark information for realizing the purpose of controlling the display, and start, stop, parse image, 1, 2, 3, 4, image save path, export video, video save path, calibration file used and modify calibration file, etc. for realizing the purpose of controlling the process. Among them, 1, 2, 3 and 4 correspond to the pictures collected by the four cameras respectively, and the other buttons are linked to the corresponding content, which can respond to the user's click operation to present the corresponding display information or start the corresponding control process. The display area includes image 1, image 2, image 3, image 4, model reasoning bird's-eye view and post-processing bird's-eye view obtained by the four cameras respectively.

[0090] S106: Based on the visualization interface of the computer, the visualization verification data is displayed.

[0091] In actual applications, when the user clicks a button on the visualization interface, the device for real-time verification of the visual model reasoning effect will respond promptly and display the visualization verification data based on the visualization interface on the computer.

[0092] In addition, since visual algorithm development engineers often have the need for offline playback on a computer, this application describes a possible recording method.

[0093] In one case, regarding how to meet the playback requirement, the method further includes:

[0094] Based on the visual interface of the computer, in response to a start recording instruction, at least one of the video stream data, the first structured data, and the second structured data is recorded to obtain recorded data;

[0095] Based on the visual interface of the computer, the recorded data is saved locally in response to a stop recording instruction.

[0096] In actual applications, the device for real-time verification of the visual model reasoning effect will also record the effect of the functional real-vehicle test and save it locally as a log. In this way, the visual algorithm development engineer can play back the log offline on the computer and view the road test effect without following the car. Specifically, the visual interface mainly includes buttons such as start recording, stop recording, select storage location, and select recording data channel. There are multiple data channels for communication, and you can select the required channel for recording to avoid unnecessary recording and space waste. Click the start button to start recording data, and click the stop button to stop recording data, forming a segmented recording.

[0097] In addition, since the methods of playing recorded videos are different, this application can describe a possible playback method.

[0098] In one case, regarding how to play a recorded video, the method further includes:

[0099] Based on the visual interface of the computer, the recorded data is parsed in response to the parsing instruction to obtain parsed image data;

[0100] Based on the visualization interface of the computer, the visualization effect is exported in video format in response to the export instruction. In practical applications, the device for real-time verification of the reasoning effect of the visual model can be based on the visualization interface of the computer, and can complete the parsing of the log in response to the parsing instruction to obtain image data, which can be used for reasoning on the model training end. In addition, the device for real-time verification of the reasoning effect of the visual model can be based on the visualization interface of the computer, and can export the visualization effect into a video and play the video data in response to the export instruction, so that the visual algorithm engineer can view the model reasoning effect on the computer.

[0101] In addition, the device for real-time verification of visual model reasoning effect provided by the present application also has the function of data re-injection. Specifically, Figure 7 A schematic diagram of a data recharge method provided by this application. Figure 7 As shown, the data re-injection function of the device for real-time verification of visual model reasoning effect provided by this application is mainly used on the test bench, which can republish the log recorded by the real vehicle, and set a subscription module in the vehicle controller to subscribe to the image video stream data recorded by the real vehicle road test, and complete the decoding to obtain the image as the input of the visual perception function module. In this way, the visual algorithm engineer can verify on the test bench whether the problems of the previous version of the software have been solved and whether other problems have been exposed without the need for real vehicle testing again.

[0102] In summary, the present application first obtains the video stream data of the target image, and processes the original image data of the target image based on the visual perception function module to obtain the first structured data. The first structured data is then supplemented to obtain the second structured data. Based on the transport layer UDP protocol, the video stream data, the first structured data and the second structured data are sent to the computer terminal in real time through serialization operations for visualization. Finally, the visualized video stream data, the first structured data and the second structured data are matched according to the timestamp to obtain the visualization verification data and display it. In this way, the real-time visualization of the visual perception function module from the vehicle controller to the computer terminal is realized based on the UDP protocol, which improves the verification efficiency of the visual model reasoning effect.

[0103] Based on the method for real-time verification of visual model reasoning effect provided in the above embodiment, the present application also provides a device for real-time verification of visual model reasoning effect. The device for real-time verification of visual model reasoning effect is described below in combination with the embodiments and drawings.

[0104] Figure 8 A schematic diagram of the structure of a device for real-time verification of visual model reasoning effect provided by this application. Figure 8 As shown, the device 200 for real-time verification of visual model reasoning effect provided in the embodiment of the present application includes:

[0105] An image encoding module 201 is used to obtain video stream data of a target image;

[0106] A first processing module 202, configured to process the original image data of the target image based on the visual perception function module to obtain first structured data;

[0107] A second processing module 203, configured to perform supplementary processing on the first structured data to obtain second structured data;

[0108] A data publishing module 204 is used to send the video stream data, the first structured data and the second structured data to the computer terminal in real time by serializing the data based on the transport layer UDP protocol;

[0109] A visualization processing module 205 is used to perform visualization processing on the video stream data, the first structured data and the second structured data based on a computer, and match the visualized video stream data, the first structured data and the second structured data according to timestamps to obtain visualization verification data;

[0110] The display module 206 is used to display the visual verification data based on the visual interface of the computer.

[0111] As an implementation method, with respect to how to obtain the video stream data of the target image, the above-mentioned apparatus 200 for real-time verification of the visual model reasoning effect further includes: an acquisition submodule;

[0112] An acquisition submodule, used for acquiring the image address of the target image based on inter-core communication;

[0113] Based on the image address, the complete image data of the target image is acquired in an image format to achieve acquisition of the target image.

[0114] As an implementation manner, with respect to how to obtain the video stream data of the target image, the above-mentioned image encoding module 201 is specifically used for:

[0115] The complete image data is compressed to obtain a video stream in a predetermined format corresponding to the complete image data, thereby acquiring the video stream data of the target image.

[0116] As an implementation manner, regarding how to obtain the first structured data, the first processing module 202 is specifically used to:

[0117] Preprocessing the original image data of the target image based on the visual perception function module to obtain first image data that meets the input of the visual perception function model;

[0118] The first image data is inferred using the visual perception function model, and first structured data is obtained through tensor processing.

[0119] As an implementation method, regarding how to send data to the computer, the data publishing module 204 is specifically used to:

[0120] Based on the transport layer UDP protocol, the first structured data is sent to the computer in real time using the first channel;

[0121] Based on the UDP protocol, the second structured data is sent to the computer terminal in real time using the second channel;

[0122] Based on the UDP protocol, the video stream data is sent to the computer terminal in real time using a third channel.

[0123] As an implementation method, with respect to how to realize data recording, the above-mentioned device 200 for real-time verification of visual model reasoning effect further includes: a real vehicle recording module;

[0124] A real vehicle recording module, for recording at least one of the video stream data, the first structured data and the second structured data based on a visual interface of a computer terminal in response to a start recording instruction, to obtain recorded data;

[0125] Based on the visual interface of the computer, the recorded data is saved locally in response to a stop recording instruction.

[0126] As an implementation method, with respect to how to play recorded data, the above-mentioned apparatus 200 for real-time verification of visual model reasoning effect further includes: an export module;

[0127] An export module, configured to parse the recorded data based on the visual interface of the computer terminal in response to a parsing instruction to obtain parsed image data;

[0128] Based on the visualization interface of the computer, the visualization effect is exported in a video format in response to an export instruction.

[0129] In summary, the present application first obtains the video stream data of the target image, and processes the original image data of the target image based on the visual perception function module to obtain the first structured data. The first structured data is then supplemented to obtain the second structured data. Based on the transport layer UDP protocol, the video stream data, the first structured data and the second structured data are sent to the computer terminal in real time through serialization operations for visualization. Finally, the visualized video stream data, the first structured data and the second structured data are matched according to the timestamp to obtain the visualization verification data and display it. In this way, the real-time visualization of the visual perception function module from the vehicle controller to the computer terminal is realized based on the UDP protocol, which improves the verification efficiency of the visual model reasoning effect.

[0130] In addition, the present application also provides a device for real-time verification of visual model reasoning effects, including: a memory for storing a computer program; a processor for implementing the steps of the method for real-time verification of visual model reasoning effects as described in any of the above items when executing the computer program.

[0131] In addition, the present application also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for real-time verification of visual model reasoning effects as described in any of the above items are implemented.

[0132] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for real-time verification of visual model reasoning effect, characterized in that: The method comprises: Get the video stream data of the target image; Processing the original image data of the target image based on the visual perception function module to obtain first structured data; Performing supplementary processing on the first structured data to obtain second structured data; Based on the transport layer UDP protocol, the video stream data, the first structured data and the second structured data are sent to the computer in real time by serialization operation; Based on the computer, respectively, the video stream data, the first structured data, and the second structured data are visualized, and the visualized video stream data, the first structured data, and the second structured data are matched according to timestamps to obtain visualization verification data; Based on the visual interface of the computer, the visual verification data is displayed.

2. The method according to claim 1, characterized in that Before acquiring the video stream data of the target image, the method further includes: Obtaining the image address of the target image based on inter-core communication; Based on the image address, the complete image data of the target image is acquired in an image format to achieve acquisition of the target image.

3. The method according to claim 2, characterized in that The step of acquiring the video stream data of the target image includes: The complete image data is compressed to obtain a video stream in a predetermined format corresponding to the complete image data, thereby acquiring the video stream data of the target image.

4. The method according to claim 1, characterized in that: The processing of the original image data of the target image based on the visual perception function module to obtain the first structured data includes: Preprocessing the original image data of the target image based on the visual perception function module to obtain first image data that meets the input of the visual perception function model; The first image data is inferred using the visual perception function model, and first structured data is obtained through tensor processing.

5. The method according to claim 1, characterized in that The video stream data, the first structured data and the second structured data are sent to the computer in real time by serialization operation based on the transport layer UDP protocol, including: Based on the transport layer UDP protocol, the first structured data is sent to the computer in real time using the first channel; Based on the UDP protocol, the second structured data is sent to the computer terminal in real time using the second channel; Based on the UDP protocol, the video stream data is sent to the computer terminal in real time using a third channel.

6. The method according to claim 1, characterized in that The method further comprises: Based on the visual interface of the computer, in response to a start recording instruction, at least one of the video stream data, the first structured data, and the second structured data is recorded to obtain recorded data; Based on the visual interface of the computer, the recorded data is saved locally in response to a stop recording instruction.

7. The method according to claim 6, characterized in that The method further comprises: Based on the visual interface of the computer, the recorded data is parsed in response to the parsing instruction to obtain parsed image data; Based on the visualization interface of the computer, the visualization effect is exported in a video format in response to an export instruction.

8. A device for real-time verification of visual model reasoning effect, characterized in that: include: An image encoding module, used for acquiring video stream data of a target image; A first processing module, used for processing the original image data of the target image based on the visual perception function module to obtain first structured data; A second processing module, configured to perform supplementary processing on the first structured data to obtain second structured data; A data publishing module, used for sending the video stream data, the first structured data and the second structured data to the computer terminal in real time by serialization operation based on the transport layer UDP protocol; A visualization processing module, for performing visualization processing on the video stream data, the first structured data and the second structured data respectively based on a computer, and matching the visualized video stream data, the first structured data and the second structured data according to timestamps to obtain visualization verification data; The display module is used to display the visual verification data based on the visual interface of the computer.

9. A device for real-time verification of visual model reasoning effect, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the method for real-time verification of visual model reasoning effect as described in any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for real-time verification of visual model reasoning effects as described in any one of claims 1 to 7.

Images