Data processing method and device, storage medium and equipment

By obtaining and encoding the camera pictures and control system data of autonomous driving vehicles in real time on the RTSP server, and serializing them and sending them through the RTSP protocol, the problem that the autonomous driving data acquisition system cannot achieve full data acquisition is solved, and automatic Efficiency of driving software development.

CN119030958APending Publication Date: 2024-11-26GAC AION NEW ENERGY AUTOMOBILE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411142184.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing autonomous driving data acquisition system cannot achieve full data acquisition, which limits the efficiency of simulation analysis and recharge testing in the development of autonomous driving software.

Method used

The pictures and control system data collected by all cameras are obtained in real time on the RTSP server, encoding and serialization are performed, and the aligned data is sent to the RTSP client through the RTSP protocol to ensure that the controller's full operating data is collected in real time.

Benefits of technology

Real-time acquisition of full operating data of the controller is realized, providing important support for offline analysis, simulation analysis and re-injection testing during the development of autonomous driving software, and improving development efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119030958A_ABST
    Figure CN119030958A_ABST
Patent Text Reader

Abstract

The invention provides a data processing method and device, a storage medium and equipment, in the method, a controller collects and compresses pictures of all cameras in real time, and collects, aligns and serializes all control system data in real time, and then, in order to ensure the accuracy of a time sequence, the data processing efficiency is improved. And the coded picture data and the serialized control system data are aligned, and then the aligned data are sent out based on an RTSP protocol. Therefore, real-time acquisition of full-amount operation data of the controller is realized, and important support is provided for offline analysis, simulation analysis and recharge test in an automatic driving software development process, so that the development efficiency is effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of automobile software development, and in particular to a data processing method, apparatus, storage medium and device. Background Art

[0002] With the continuous advancement of automobile technology, autonomous driving is increasingly being used in people's daily driving. For the functional development of autonomous driving control, collecting full and real-time controller data is one of the most important development methods. The full amount of data can be used to observe whether the function is normal, and it is also a necessary prerequisite for simulation analysis and re-injection testing. However, in the field of autonomous driving, the current data acquisition system generally only supports the collection of specified data and cannot achieve full data collection. Summary of the invention

[0003] The purpose of this application is to provide a data processing method, device, storage medium and equipment, aiming to solve the problem that the autonomous driving data collection method in the related technology cannot realize full data collection.

[0004] In a first aspect, the present application provides a data processing method, which is applied to an RTSP server, and the RTSP server runs on a controller of a vehicle; the method comprises: acquiring pictures and control system data collected by all cameras in real time, encoding the pictures to obtain a media data stream, and aligning and serializing the control system data to obtain serialized data; aligning the media data stream and the serialized data; and after completing a handshake with the RTSP client, sending the aligned media data stream and serialized data to the RTSP client.

[0005] In the above implementation process, the controller collects and compresses the images of all cameras in real time, and collects, aligns and serializes all control system data in real time. Afterwards, in order to ensure the accuracy of the timing, the encoded image data and the serialized control system data are aligned, and then the aligned data is sent out based on the RTSP protocol. In this way, the real-time collection of the full amount of running data of the controller is realized, which provides important support for offline analysis, simulation analysis, and re-injection testing in the process of autonomous driving software development, thereby effectively improving development efficiency.

[0006] Furthermore, in some examples, the media data stream is obtained by encoding the image through an encoder; before the real-time acquisition of the images collected by all cameras and the control system data, it includes: performing an initialization configuration operation when the controller is initialized or before the acquisition program of the controller is started; the initialization configuration operation includes initializing the parameters of the encoder, controlling the encoder to enter the working mode, and initializing the parameters of the RTSP server, and starting the target thread of the RTSP server.

[0007] In the above implementation process, when the controller is initialized or before the acquisition program is started, the encoder and RTSP server are initialized and configured to ensure smooth data processing and improve the reliability of data acquisition.

[0008] Furthermore, in some examples, the parameters of the encoder include: the size and format of the input image, the format and frame rate of the media data stream, the encoding mode, GOP, cropping parameters and ROI parameters; the parameters of the RTSP server include: the number of channels, channel attributes, the source of the image and the source of the control system data.

[0009] In the above implementation process, the accuracy and efficiency of data collection can be improved by initializing the encoder parameters and server parameters.

[0010] Furthermore, in some examples, the aligning and serializing the control system data includes: aligning the control system data using a target flag; and performing a ProtoBuf serialization operation on the aligned control system data.

[0011] In the above implementation process, a specific method for aligning and serializing the input and output signals of each software module of the controller is provided.

[0012] Furthermore, in some examples, aligning the media data stream and the serialized data includes: caching the media data stream and the serialized data, and aligning the media data stream and the serialized data in a cache area.

[0013] In the above implementation process, the cache area is used as a data buffer, the encoded image data and the serialized control system data are temporarily stored in the cache area, and alignment work is performed in the cache area, so that the alignment process of the media data stream and the serialized data is smoother and the data processing flow is optimized.

[0014] Furthermore, in some examples, the aligning of the media data stream and the serialized data in the cache area includes: in the cache area, for each frame of the media data stream, matching data generated by reasoning, post-processing, tracking and / or fusion based on the image in the serialized data, and aligning the acquisition time of the data with the acquisition time of the image.

[0015] In the above implementation process, the acquisition time of the image is used as a unified time axis to achieve time synchronization between the image and the data matching it, thereby solving the problem of timing disorder caused by image encoding and control system data alignment serialization.

[0016] Furthermore, in some examples, the RTSP client runs on a PC host computer carried on the vehicle.

[0017] In the above implementation process, combined with a host computer that supports the RTSP protocol, real-time collection of the controller's full operating data is achieved. The host computer receives and stores the controller data. In this way, the controller data can be replayed, re-injected, and simulated in an offline manner, making it easier for testers and developers to discover, analyze, and solve problems.

[0018] In the second aspect, a data processing device provided by the present application is applied to an RTSP server, and the RTSP server runs on a controller of a vehicle; the device includes: an acquisition module, which is used to acquire pictures and control system data collected by all cameras in real time, encode the pictures to obtain media data streams, and align and serialize the control system data to obtain serialized data; an alignment module, which is used to align the media data stream and the serialized data; and a sending module, which is used to send the aligned media data stream and serialized data to the RTSP client after completing a handshake with the RTSP client.

[0019] In a third aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium, on which instructions are stored. When the instructions are executed on a computer, the computer executes the method as described in any one of the first aspects.

[0021] In a fifth aspect, the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method as described in any one of the first aspects.

[0022] Other features and advantages disclosed in the present application will be described in the following description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology disclosed in the present application.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the workflow of a vehicle-side real-time data collection solution based on the RTSP protocol provided in an embodiment of the present application;

[0027] Figure 3 A block diagram of a data processing device provided in an embodiment of the present application;

[0028] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0030] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0031] As described in the background technology, the data collection method for autonomous driving in the related art has the problem of being unable to collect full data. Based on this, the embodiment of the present application provides a data processing solution to solve the above problem.

[0032] Next, the embodiments of the present application are introduced:

[0033] like Figure 1 As shown, Figure 1 It is a flow chart of a data processing method provided by an embodiment of the present application. The method is applied to an RTSP server, and the RTSP server runs on a controller of a vehicle. That is to say, the scheme of this embodiment runs on the controller of the vehicle, and the controller may be an automatic driving controller, which is responsible for integrating data from various sensors of the vehicle and making driving decisions according to a pre-set algorithm, thereby controlling the steering, acceleration and braking of the vehicle. Of course, the controller may also be other types of controllers.

[0034] In this embodiment, the controller establishes an RTSP (Real Time Streaming Protocol) connection with the host computer. RTSP is an application layer protocol for controlling real-time streaming media transmission. The protocol has the following characteristics: supports real-time transmission, can achieve fast and instant streaming media transmission; has good scalability and network adaptability; provides rich control functions, etc. The controller implements data outbound transmission based on the RTSP protocol, so that the host computer can collect the full amount of running data of the controller in real time.

[0035] The method comprises:

[0036] Step 101: Acquire pictures and control system data collected by all cameras in real time, encode the pictures to obtain media data streams, and align and serialize the control system data to obtain serialized data;

[0037] The pictures and control system data mentioned in this step can constitute the full operating data of the controller. The type of pictures here mainly depends on the type and position of the camera and the specific configuration of the vehicle. For example, the pictures collected by the front camera can be used to indicate the road conditions in front of the vehicle, including lane lines, traffic signs, vehicles in front, etc.; the pictures collected by the rearview camera can be used to indicate the road conditions behind the vehicle; the pictures collected by the interior camera can be used to monitor driver behavior and passenger status, etc. The control system data here can include CAN (Controller Area Network) bus data, such as radar sensor detection data, chassis signals, etc., and can also include ETH (Ethernet) bus data and important data inside the controller. These data can help developers locate, analyze and test problems more efficiently.

[0038] In this embodiment, the controller encodes the image and converts it into a format suitable for streaming media transmission, such as H.264, H.265 and other video encoding formats, so as to be transmitted through the RTSP protocol. The encoded image data is the media data stream mentioned in this step, and the media data stream can be obtained by encoding the image through an encoder. The encoder here can be a hardware encoder or a software encoder. When the hardware of the controller supports hardware encoding, hardware encoding is usually preferred to reduce the occupation of computing resources.

[0039] At the same time, the application software of the controller is usually divided into multiple software modules, and the scheduling timing and cycle of the software modules are diversified and personalized. Therefore, the controller aligns the input and output signals of each software module, that is, the control system data, to achieve synchronization of timing and logic. After aligning the data, the data is serialized, which can solve the differences in multiple languages ​​and compiled software and hardware platforms, so that the serialized data can be used across platforms and languages, and can also solve some special data structures in the application software, such as non-fixed-length arrays, special container types, etc. In some embodiments, the alignment and serialization of the control system data mentioned in this step may include: aligning the control system data using a target flag; performing a ProtoBuf serialization operation on the aligned control system data. The target flag here can be a picture timestamp, a picture frame number, a system time, or a specific flag signal of an application, etc., and the control system data can be aligned using the target flag. ProtoBuf is the abbreviation of Protocol Buffers, which is a data description language that provides a lightweight and efficient way to serialize and deserialize structured data. During implementation, it is necessary to add the ProtoBuf library in the compilation environment and running environment of the controller in advance. For specific ProtoBuf serialization operations, please refer to the introduction of ProtoBuf in the relevant technology, and this application does not limit this.

[0040] In some embodiments, this step may include: performing an initialization configuration operation when the controller is initialized or before the acquisition program of the controller is started; the initialization configuration operation includes initializing the parameters of the encoder, controlling the encoder to enter the working mode, and initializing the parameters of the RTSP server, and starting the target thread of the RTSP server. That is, before acquiring the image and controlling the system data, the encoder and the RTSP server need to be initialized and configured to ensure that the data processing process proceeds smoothly and improve the reliability of data acquisition. In implementation, the initialization configuration operation can be performed immediately when the controller is initialized, or before the acquisition program is started, wherein the initialization configuration operation for the encoder includes initializing the parameters of the encoder and allowing the encoder to enter the working mode; accordingly, the initialization configuration operation for the RTSP server includes initializing the parameters of the RTSP server and starting the target thread of the RTSP server. The target thread here can refer to a dedicated thread of the RTSP server, that is, a thread created specifically for processing RTSP protocol-related tasks.

[0041] Optionally, the parameters of the encoder may include: the size and format of the input image, the format and frame rate of the media data stream, the encoding mode, GOP (Group Of Pictures), cropping parameters, and ROI (Region Of Interest) parameters; the parameters of the RTSP server may include: the number of channels, channel attributes, the source of the image, and the source of the control system data. By initializing these parameters, the accuracy and efficiency of data acquisition can be improved.

[0042] Step 102: aligning the media data stream and the serialized data;

[0043] This step means: in actual applications, usually, images are pushed to various application software to run and generate data only after they are generated, that is, the images are generated before the data. However, since the encoding or compression of images is an asynchronous operation, the time consumption may be uncertain. Therefore, there is no strict sequence between the media data stream and the serialized data. Therefore, the media data stream and the serialized data need to be realigned to ensure signal consistency.

[0044] In some embodiments, this step may include: caching the media data stream and the serialized data, and aligning the media data stream and the serialized data in the cache area. That is, the controller may use the cache area as a data buffer, temporarily store the encoded image data and the serialized control system data in the cache area, and perform alignment work in the cache area, so that the alignment process of the media data stream and the serialized data is smoother and the data processing flow is optimized.

[0045] Furthermore, the aforementioned alignment of the media data stream and the serialized data in the cache area may include: in the cache area, for each frame of the media data stream, matching the data generated based on the reasoning, post-processing, tracking and / or fusion of the image in the serialized data, and aligning the acquisition time of the data with the acquisition time of the image. In other words, the data based on the reasoning, post-processing, tracking, and fusion of the image needs to be aligned and matched with the image, and the controller uses the acquisition time of the image as a unified time axis to achieve time synchronization between the image and the data matching it, thereby solving the problem of timing disorder caused by image encoding and control system data alignment serialization.

[0046] Step 103: After completing the handshake with the RTSP client, the aligned media data stream and serialized data are sent to the RTSP client.

[0047] This step means that after the RTSP server and the RTSP client complete the handshake, the RTSP server can place the aligned media data stream and serialized data in the cache in the connection channel and send them to the RTSP client in real time, thereby completing the collection and sending of controller data.

[0048] In some embodiments, the RTSP client mentioned in this step runs on the PC host computer on the vehicle. That is to say, unlike the traditional vehicle-cloud collaboration, in this embodiment, the full running data of the controller can be transmitted to the PC (Personal Computer) host computer on the vehicle based on the RTSP protocol. The PC host computer receives and stores the full running data of the controller. In this way, the controller data can be replayed, refilled, and simulated in an offline manner, which is more convenient for testers and developers to find, analyze, and solve problems.

[0049] In the embodiment of the present application, the controller collects and compresses the images of all cameras in real time, and collects, aligns and serializes all control system data in real time. Afterwards, in order to ensure the accuracy of the timing, the encoded image data and the serialized control system data are aligned, and then the aligned data is sent out based on the RTSP protocol. In this way, the real-time collection of the full amount of operating data of the controller is realized, which provides important support for offline analysis, simulation analysis, and re-injection testing in the process of autonomous driving software development, thereby effectively improving development efficiency.

[0050] In order to explain the solution of this application in more detail, a specific embodiment is introduced below:

[0051] This embodiment provides a vehicle-side real-time data acquisition solution based on the RTSP protocol. The solution is run on the controller, and its compilation environment, development language, operating system and third-party library should be selected according to the actual situation of the controller's SoC (System on Chip) chip and project development. During implementation, an RTSP server runs on the controller, and an RTSP client runs on the controller's host computer.

[0052] The workflow of this scheme is as follows Figure 2 As shown, including:

[0053] S201. Add dependent third-party libraries;

[0054] Specifically, in order to implement the solution of this embodiment, it is necessary to add dependent third-party libraries, including RTSPServer protocol library and ProtoBuf library, in the compilation environment and running environment of the controller. If software coding is used, it is also necessary to add a software coding library; the version of the third-party library needs to be compatible with the operating system of the controller chip and the model or version of the compiler;

[0055] S202, initializing configuration of the encoder and RTSP server;

[0056] Specifically, when the controller is initialized or before the acquisition program is started, the encoder and RTSP server are initialized and configured, wherein the initialization configuration of the encoder includes: setting the size and format of the input image, the format and frame rate of the output media data stream, the encoding mode, GOP, cropping parameters and ROI parameters, etc., and controlling the encoder to enter the working mode; the initialization configuration of the RTSP server includes: setting the number of channels, channel attributes, sources and cache areas of the server's image and control system data, and starting the dedicated thread of the RTSP server, waiting for the client to handshake and connect;

[0057] S203, obtaining images from all cameras and encoding them to obtain a media data stream;

[0058] Specifically, when the controller's hardware supports hardware encoding, hardware encoding is usually preferred to reduce the occupation of chip computing resources by software encoding code. For the hardware encoding method, the software first initializes the hardware encoder with the correct parameters. Then, when the software subscribes to obtain a new image (for example, a YUV format image), the image should be sent to the hardware encoder in a timely manner, and wait for the hardware encoder to output the media data stream (or jpg format image). In addition, in order to achieve synchronization between the control system data and the media data stream, the output media data stream needs to be marked with information such as timestamps;

[0059] S204, collecting control system data and performing alignment and serialization processing to obtain serialized data;

[0060] Specifically, the application software of the controller is usually divided into multiple software modules. The input and output signals of these software modules are control system data, including CAN bus data, ETH bus data, and important data inside the controller. Since the scheduling timing and cycle of the software modules are diverse and personalized, it is necessary to synchronize and align the input and output signals of each software module. When implementing, the timing and logic synchronization, alignment and integrity check can be performed by using image timestamps, image frame numbers, system time, or specific flag signals of the application program;

[0061] After the acquisition program has completely collected and aligned the control system data, the control system data needs to be serialized using ProtoBuf. The purpose of serialization is, on the one hand, to resolve the differences between multiple languages ​​and compiled software and hardware platforms, so that the serialized data can be used across platforms and languages, facilitating the platformization of simulation, playback, and re-injection, and on the other hand, to resolve some special data structures in the application, including non-fixed-length arrays, special container types, etc.

[0062] In addition, there is no fixed order between this step and S203, and the two can be performed simultaneously;

[0063] S205, caching and aligning the media data stream and serialized data;

[0064] Specifically, usually, images are pushed to various application software to run and generate data only after they are generated. That is, images are generated before data. However, since the encoding or compression of images is an asynchronous operation, its time consumption may be uncertain. Therefore, there is no strict sequence between the media data stream and serialized data after image encoding. Therefore, the media data stream and serialized data need to be cached and aligned again.

[0065] During implementation, the media data stream and serialized data are cached, and the images and data are synchronized and aligned in the cache area, so that the data for inference, post-processing, tracking, and fusion based on the image are aligned and matched with the image.

[0066] S206, RTSP server data is sent out;

[0067] Specifically, after the RTSP client and the RTSP server complete the handshake, the RTSP server places the aligned media data streams and serialized data in the cache in the corresponding channels and sends them to the RTSP client in real time, thereby completing the collection and transmission of controller data.

[0068] This embodiment scheme, combined with a host computer that supports the RTSP protocol, can collect and display the full amount of operating data of the controller in real time, analyze the software operating logic of the controller, and the collected data can be used for offline playback analysis problems, can be used for model simulation testing, can be used for various scenarios such as data injection, thereby providing important support for autonomous driving software development and testing, and effectively improving development efficiency.

[0069] Corresponding to the embodiments of the aforementioned method, the present application also provides embodiments of a data processing device and a terminal using the same:

[0070] like Figure 3 As shown, Figure 3 : is a block diagram of a data processing device provided in an embodiment of the present application, wherein the device is applied to an RTSP server, and the RTSP server runs on a controller of a vehicle; the device comprises:

[0071] The acquisition module 31 is used to acquire the pictures and control system data collected by all cameras in real time, encode the pictures to obtain media data streams, and align and serialize the control system data to obtain serialized data;

[0072] An alignment module 32, used for aligning the media data stream and the serialized data;

[0073] The sending module 33 is used to send the aligned media data stream and serialized data to the RTSP client after completing the handshake with the RTSP client.

[0074] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0075] This application also provides an electronic device, see Figure 4 , Figure 4A block diagram of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 410, a communication interface 420, a memory 430, and at least one communication bus 440. The communication bus 440 is used to realize direct connection and communication between these components. The communication interface 420 of the electronic device in the embodiment of the present application is used to communicate signaling or data with other node devices. The processor 410 may be an integrated circuit chip with signal processing capabilities.

[0076] The processor 410 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor 410 can also be any conventional processor, etc.

[0077] The memory 430 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 430 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 410, the electronic device can execute the above-mentioned Figure 1 The method embodiment involves various steps.

[0078] Optionally, the electronic device may further include a storage controller and an input / output unit.

[0079] The memory 430, storage controller, processor 410, peripheral interface, input and output unit components are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 440. The processor 410 is used to execute executable modules stored in the memory 430, such as software function modules or computer programs included in the electronic device.

[0080] The input and output unit is used to provide users with the task creation and to create a start optional time period or preset execution time for the task to realize the interaction between the user and the server. The input and output unit can be, but is not limited to, a mouse and a keyboard.

[0081] Understandably, Figure 4 The structure shown is for illustration only, and the electronic device may also include Figure 4 More or fewer components as shown, or with Figure 4 Different configurations shown. Figure 4 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0082] An embodiment of the present application further provides a storage medium having instructions stored thereon. When the instructions are run on a computer, the computer program is executed by a processor to implement the method described in the method embodiment. To avoid repetition, the method will not be described here.

[0083] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0084] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0085] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0086] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0087] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0088] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0089] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims

1. A data processing method, characterized in that: Applied to an RTSP server, the RTSP server runs on a controller of a vehicle; the method comprises: Acquire the pictures and control system data collected by all cameras in real time, encode the pictures to obtain media data streams, and align and serialize the control system data to obtain serialized data; Aligning the media data stream with the serialized data; After completing the handshake with the RTSP client, the aligned media data stream and serialized data are sent to the RTSP client.

2. The method according to claim 1, characterized in that The media data stream is obtained by encoding the picture through an encoder; Before obtaining the pictures and control system data collected by all cameras in real time, the following steps are included: When the controller is initialized, or before the acquisition program of the controller is started, an initialization configuration operation is performed; the initialization configuration operation includes initializing the parameters of the encoder, controlling the encoder to enter the working mode, initializing the parameters of the RTSP server, and starting the target thread of the RTSP server.

3. The method according to claim 2, characterized in that The parameters of the encoder include: the size and format of the input picture, the format and frame rate of the media data stream, the encoding mode, GOP, cropping parameters and ROI parameters; The parameters of the RTSP server include: the number of channels, channel attributes, the source of the pictures, and the source of the control system data.

4. The method according to claim 1, characterized in that: The aligning and serializing the control system data includes: aligning the control system data using a target flag; Perform ProtoBuf serialization operation on the aligned control system data.

5. The method according to claim 1, characterized in that The aligning the media data stream and the serialized data includes: The media data stream and the serialized data are cached, and the media data stream and the serialized data are aligned in a cache area.

6. The method according to claim 5, characterized in that The aligning the media data stream and the serialized data in the cache area includes: In the cache area, for each frame of the image in the media data stream, data generated by reasoning, post-processing, tracking and / or fusion based on the image is matched in the serialized data, and the acquisition time of the data is aligned with the acquisition time of the image.

7. The method according to claim 1, characterized in that The RTSP client runs on a PC host computer carried on the vehicle.

8. A data processing device, characterized in that: Applied to an RTSP server, the RTSP server runs on a controller of a vehicle; the device comprises: An acquisition module is used to acquire the images and control system data collected by all cameras in real time, encode the images to obtain media data streams, and align and serialize the control system data to obtain serialized data; An alignment module, used for aligning the media data stream and the serialized data; The sending module is used to send the aligned media data stream and serialized data to the RTSP client after completing the handshake with the RTSP client.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.