Data processing method and apparatus

Abstract

Embodiments of the present application provide a data processing method and device, the method comprising: a second processing unit acquires first data from a first processing unit, pre-processes the first data to obtain to-be-processed data; inputs the to-be-processed data into a processing module to obtain a processing result, and waits for the first processing unit to call, wherein the processing module contains an algorithm model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and to, but is not limited to, a data processing method and apparatus. Background Technology

[0002] In the context of connected vehicles, there are high requirements for video latency, which is the most important factor in the video processing process.

[0003] When processing video, the large amount of data after decoding makes both preprocessing and neural network inference time-consuming. To address this issue, some technologies place preprocessing and neural network inference entirely on the central processing unit (CPU), or accelerate some inference processes on the graphics processing unit (GPU). However, these methods fail to meet the requirements for low latency. Summary of the Invention

[0004] Based on the problems existing in related technologies, this application provides a data processing method and apparatus.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a data processing method, the method comprising:

[0007] The second processing unit obtains the first data from the first processing unit, preprocesses the first data, and obtains the data to be processed.

[0008] The data to be processed is input into the processing module to obtain the processing result, which is then retrieved by the first processing unit. The processing module contains an algorithm model.

[0009] In some embodiments, if the processing module includes multiple processing sub-modules, the data to be processed is processed in each of the processing sub-modules using a pipelined parallel approach.

[0010] In some embodiments, the preprocessing includes at least: data decoding;

[0011] Before preprocessing the first data, the method further includes: initializing the video memory pool based on the first data.

[0012] In some embodiments, the first data is first video data.

[0013] The data to be processed obtained after the first video data is decoded is the image to be processed.

[0014] The initialization of the video memory pool based on the first data includes:

[0015] The size of the video memory in the video memory pool is determined based on the size of the first video data, and the size of the video memory corresponds to the size.

[0016] In some embodiments, the preprocessing further includes: scaling processing;

[0017] After obtaining the image to be processed, the method further includes: scaling the size of the image to be processed so that the scaled size meets the size requirements of the processing submodule to be input.

[0018] In some embodiments, the processing submodule includes a target detection module, a feature detection module, a target tracking module, and a data fusion module;

[0019] Accordingly, the data to be processed is input into the processing module to obtain the processing result, including:

[0020] The image to be processed is sequentially input into the target detection module, the feature detection module, the target tracking module, and the data fusion module to obtain the processing result.

[0021] In some embodiments, the step of sequentially inputting the image to be processed into the target detection module, the feature detection module, the target tracking module, and the data fusion module to obtain the processing result includes:

[0022] The image to be processed is input into the target detection module, the target detection is performed on the image to be processed, a detection image with the target object is obtained, and the detection image is sent to the feature detection module;

[0023] The target features are extracted from the detection image in the feature detection module to determine the feature information of the target detection object, thereby obtaining a feature image with feature information, and the feature image is sent to the target tracking module.

[0024] Based on the feature information, target tracking is performed on the target detection object in the feature image of the target tracking module to obtain target tracking information, and the tracked image with the target tracking information is sent to the data fusion module.

[0025] The target tracking information and the feature information of the target detection object in the tracking image in the data fusion module are fused to obtain a target image with target detection object fusion information.

[0026] The target image containing the target detection object fusion information is determined as the processing result.

[0027] In some embodiments, performing target detection on the image to be processed to obtain a detected image with the target object includes:

[0028] Target detection is performed on the image to be processed to obtain at least one target detection object and at least one detection box corresponding to each target detection object;

[0029] The confidence score of each detection box is calculated by using the non-maximum suppression algorithm.

[0030] The detection box with the highest confidence score is determined as the target detection box corresponding to each target detection object.

[0031] The image to be processed containing the target detection box is determined as a detection image containing the target object.

[0032] In some embodiments, the step of extracting target features from the detection image in the feature detection module, determining the feature information of the target detection object, and obtaining a feature image with feature information includes:

[0033] Based on the target detection box corresponding to the target detection object, the target detection object in the detection image is cropped sequentially to obtain the cropped image and the image size corresponding to the cropped image;

[0034] Feature extraction is performed on the target detection object in the cropped image to obtain the location and category of the target detection object;

[0035] The detected image having the specified location and category is identified as the feature image.

[0036] This application provides a data processing apparatus, the apparatus comprising:

[0037] An acquisition unit is used for the second processing unit to acquire first data from the first processing unit, preprocess the first data, and obtain data to be processed.

[0038] The processing unit is used to input the data to be processed into the processing module to obtain the processing result, which is then retrieved by the first processing unit. The processing module contains an algorithm model.

[0039] The data processing device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the data processing method described in this application.

[0040] The computer-readable storage medium provided in this application embodiment stores executable instructions thereon, which are used to cause a processor to execute the executable instructions to implement the data processing method provided in this application embodiment.

[0041] This application provides a computer program product, which includes executable instructions stored in a computer-readable storage medium; when a processor of a data processing device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the above-described data processing method is implemented.

[0042] The data processing method and apparatus provided in this application embodiment acquire first data from a first processing unit through a second processing unit, preprocess the first data to obtain data to be processed, and input the data to be processed into a processing module to obtain a processing result, which is then retrieved by the first processing unit. This allows the second processing unit to perform data preprocessing and AI inference, avoiding multiple copies of data between the second and first processing units, thus improving data processing efficiency, saving bandwidth, and reducing data latency. Attached Figure Description

[0043] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method provided in the embodiments of this application;

[0044] Figure 2 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the pipeline parallel mode provided in the embodiments of this application;

[0046] Figure 4 This is an optional flowchart illustrating the data processing method provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of the decoding module and processing module provided in the embodiments of this application;

[0048] Figure 6 This is a schematic diagram of the video decoding process provided in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the target detection process provided in the embodiments of this application;

[0050] Figure 8 This is a schematic diagram of the feature extraction process provided in an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of the target tracking process provided in an embodiment of this application;

[0052] Figure 10 This is a schematic diagram of the data fusion process provided in an embodiment of this application;

[0053] Figure 11 This is a schematic diagram of the composition structure of the data processing device provided in the embodiments of this application;

[0054] Figure 12 This is a schematic diagram of the composition structure of the data processing device provided in the embodiments of this application. Detailed Implementation

[0055] To more clearly illustrate the purpose, technical solutions, and advantages of the embodiments of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the following description of the embodiments is intended to explain and illustrate the overall concept of the embodiments of this application, and should not be construed as a limitation of the embodiments of this application. In the specification and drawings, the same or similar reference numerals refer to the same or similar parts or components. For clarity, the drawings are not necessarily drawn to scale, and some well-known parts and structures may be omitted from the drawings.

[0056] In some embodiments, unless otherwise defined, the technical or scientific terms used in the embodiments of this application shall have the ordinary meaning understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The word "a" or "an" does not exclude multiple components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," "right," "top," or "bottom" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. When an element such as a layer, film, region, or substrate is referred to as being "above" or "below" another element, the element may be "directly" located "above" or "below" the other element, or there may be intermediate elements present.

[0057] In the context of connected vehicles, the AI ​​processing flow for video involves retrieving the video stream, video decoding, AI processing, video encoding, and push. Due to the large amount of data after video decoding, both preprocessing and neural network inference are time-consuming. Related technologies perform the entire AI processing flow on the CPU or accelerate some network inference processes on the GPU, but neither fully meets the low-latency requirements. Based on the problems in these technologies, this application provides a data processing method. A second processing unit acquires first data from a first processing unit, preprocesses the first data to obtain data to be processed, and inputs the data to be processed into a processing module to obtain a processing result, which is then retrieved by the first processing unit. This embodiment of the application uses a second processing unit to preprocess and perform AI inference on the data, avoiding multiple copies of data between the second and first processing units, thus improving data processing efficiency, saving bandwidth, and reducing data latency.

[0058] The data processing method provided in this application embodiment can be executed by electronic devices such as data processing equipment. These electronic devices can be various types of terminals, including laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or they can be implemented as servers. A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0059] The following will describe an exemplary application of the data processing device as a server. The technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the accompanying drawings.

[0060] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method provided in this application embodiment. The data processing system 10 provided in this application embodiment includes a first processing unit 100, a network 200, and a second processing unit 300. When performing data processing, the second processing unit 300 can use the method provided in this application embodiment to obtain first data from the first processing unit 100 through the network 200, preprocess the first data to obtain data to be processed, input the data to be processed into the processing module in the second processing unit 300, obtain the processing result, and wait for the first processing unit to retrieve it.

[0061] See Figure 2, Figure 2 This is an optional flowchart illustrating the data processing method provided in this application embodiment. The data processing method provided in this application embodiment can be implemented through steps S201 to S203:

[0062] Step S201: The second processing unit obtains the first data from the first processing unit, preprocesses the first data, and obtains the data to be processed.

[0063] Here, the second processing unit can be a GPU, and the first processing unit can be a CPU. The GPU obtains the first data from the CPU, which can be video data.

[0064] In some embodiments, preprocessing the first data can be performed by data decoding, which can be implemented through a decoding module. Here, hardware decoding can be used to accelerate data decoding. Hardware decoding is a decoding process implemented in hardware, where the GPU performs the decoding of video data, resulting in very low CPU utilization.

[0065] In some embodiments, before decoding the first video data, the video memory pool needs to be initialized based on the first video data. That is, the video memory size corresponding to the size of the first video data is determined in the video memory pool. Here, the first video data is obtained, an instance is created based on the first video data, and each instance corresponds to a video memory pool. The dimensions (width and height) of this video stream are parsed, and the video memory pool is initialized. For example, the first video data A is retrieved, an instance A is created, and the width and height of each frame of the video in instance A are parsed (here, each frame in the video has the same dimensions). A portion of the video memory is taken from the video memory as the video memory pool for instance A, and the size of each block of video memory in this pool is width × height × number of channels (where RGB images have three channels and CMYK images have four channels). For example, if the width and height of the video in instance A are 1920 × 1080, and the number of channels is 3, then the size of each block of video memory in the video memory pool of instance A is 1920 × 1080 × 3.

[0066] After initializing the video memory pool, the resource execution environment is registered with the processing unit. The processing module constructs an independent execution environment for each channel of first video data and creates a mapping relationship between the execution environment and the processing module. After video decoding, the decoded video is sent to the processing module according to this mapping relationship. Here, the processing module includes multiple processing sub-modules, which process the data to be processed in each sub-module using a pipelined parallel approach. The processing module can also be an artificial intelligence (AI) inference module.

[0067] In some embodiments, after the GPU acquires the first video data from the CPU, it places the video data into a data decoding queue. Here, the acquired first video data is compressed data. After the processing module constructs an independent execution environment for the first video data, it retrieves the first video data from the data decoding queue, performs hardware decoding to accelerate decoding, and obtains the image to be processed. The image to be processed is then identified as the data to be processed obtained after data decoding.

[0068] In some embodiments, after obtaining the image to be processed by data decoding, it is determined whether there is free video memory in the video memory pool. If there is free video memory, a video memory block corresponding to the size of the first video data is retrieved to store the decoded image to be processed, and the video memory and execution environment containing the image to be processed are encapsulated. The encapsulated data is then placed into the queue of the processing module.

[0069] In some embodiments, after the encapsulated data is placed into the queue of the processing module, the previously registered resource execution environment and mapping relationship can be deregistered from the processing module.

[0070] In some embodiments, preprocessing can also be scaling. After obtaining the image to be processed, the size of the image to be processed is scaled so that the scaled size meets the size requirements of the processing submodule to be input. For example, if the processing submodule requires the input image to be processed to be 1920×1080, then after obtaining the image to be processed, the size of the image to be processed needs to be scaled so that the size of the image to be processed is 1920×1080.

[0071] Step S202: Input the data to be processed into the processing module to obtain the processing result, which is then retrieved by the first processing unit. The processing module contains an algorithm model.

[0072] In some embodiments, the processing module includes multiple processing sub-modules, and the data to be processed is processed in each processing sub-module in a pipelined parallel manner. Here, the data to be processed can also be processed according to the priority of the data to be processed, or the data to be processed in each processing sub-module can be processed serially.

[0073] In some embodiments, taking target detection and tracking in the Internet of Vehicles as an example, the processing submodule includes a target detection module, a feature detection module, a target tracking module, and a data fusion module. The data to be processed can be input into the processing module to obtain the processing result. Alternatively, the image to be processed can be input into the target detection module, the feature detection module, the target tracking module, and the data fusion module in sequence to obtain the processing result.

[0074] The processing module provided in this application embodiment can not only perform target detection and tracking in vehicle networking scenarios, but also perform other AI inferences. This application embodiment does not impose any limitations.

[0075] It should be noted that the object detection module can be used to detect objects in the data to be processed, that is, to detect objects in the images to be processed. Object detection can refer to detecting vehicles in the images to be processed. The feature detection module is used to extract features from vehicles in the images to be processed to determine the vehicle's feature information. The object tracking module is used to continuously track vehicles in the images to be processed. If a vehicle is consistently identified from all the images to be processed in the first video data, then the target tracking information of that vehicle is determined. The data fusion module is used to fuse the target tracking information and the vehicle's feature information, and then annotate the fused information on the target image.

[0076] In some embodiments, the target detection module, feature detection module, target tracking module, and data fusion module can process the data to be processed in a pipelined parallel manner. In this embodiment, the pipelined parallel manner is used to make each processing sub-module run synchronously (that is, the data to be processed is put into the corresponding queue, and each part is run synchronously), so as to accelerate the entire data processing chain. Figure 3 This is a schematic diagram of the pipeline parallel mode provided in the embodiments of this application, such as... Figure 3 As shown, A represents instance A, and the subscript value n represents the nth frame image in instance A. Image frames in instance A undergo target detection sequentially, such as A1, A2, A3, A4, A5, A6, A7. After target detection is completed for one frame, the image is sent to the feature extraction queue. Feature extraction is performed on each frame sequentially, followed by target tracking and target fusion. Thus, this embodiment of the application uses a pipelined parallel approach to synchronize the various processing submodules, improving data processing efficiency.

[0077] This embodiment of the application obtains first data from the first processing unit through a second processing unit, preprocesses the first data to obtain data to be processed, and inputs the data to be processed into a processing module to obtain a processing result, which is then retrieved by the first processing unit. This embodiment of the application uses the second processing unit to preprocess the data and perform AI inference, avoiding multiple copies of data between the second and first processing units, thus improving data processing efficiency, saving bandwidth, and reducing data latency. Furthermore, this embodiment of the application uses hardware decoding and data preprocessing to process the data to be processed, further improving data processing efficiency. Simultaneously, this embodiment of the application effectively controls GPU resources through a memory pool, saving time on memory allocation and release, reducing time consumption, and improving efficiency.

[0078] In some embodiments, the present application embodiments can determine the sentence type of each sentence in the text to be processed by the similarity between each sentence and the set of whole sentences or the set of sentence segments. Figure 4This is an optional flowchart illustrating the data processing method provided in this application embodiment. The image to be processed is sequentially input into the target detection module, the feature detection module, the target tracking module, and the data fusion module. The processing result can be obtained through steps S401 to S405.

[0079] Step S401: Input the image to be processed into the target detection module, perform target detection on the image to be processed, obtain a detection image with the target object, and send the detection image to the feature detection module.

[0080] In some embodiments, the decoded, preprocessed, and encapsulated image to be processed is placed in the queue of the target detection module in the processing module. The target detection module obtains the image to be processed from the queue, performs target detection on the image to be processed, obtains a detection image with the target object, and sends the detection image to the queue of the feature detection module.

[0081] In some embodiments, the image to be processed can be implemented through steps S4011 to S4014:

[0082] Step S4011: Perform target detection on the image to be processed to obtain at least one target detection object and at least one detection box corresponding to each target detection object.

[0083] In some embodiments, the target detection object can be a vehicle, pedestrian, animal or other target detection object in the image to be processed. When performing target detection, each target detection object corresponds to at least one detection box. Among the multiple detection boxes corresponding to the same target detection object, each detection box has a different focus. For example, a detection box contains the entire target detection object, that is, the detection box completely covers the vehicle; a detection box contains the license plate of the target detection object, that is, the detection box only covers the license plate of the vehicle.

[0084] Step S4012: Using the non-maximum suppression algorithm, calculate the confidence level of at least one detection box corresponding to each target detection object to obtain the confidence level value of each detection box.

[0085] Step S4013: The detection box with the highest confidence value is determined as the target detection box corresponding to each target detection object.

[0086] In this embodiment, a non-maximum suppression algorithm can be used to calculate the confidence value of each detection box corresponding to the target detection object, and the detection box with the highest confidence value can be determined as the target detection box corresponding to the target detection object.

[0087] Step S4014: Determine the image to be processed containing the target detection box as a detection image containing the target detection object.

[0088] Step S402: Extract target features from the detection image in the feature detection module, determine the feature information of the target detection object, obtain a feature image with feature information, and send the feature image to the target tracking module.

[0089] In some embodiments, target feature extraction of the detected image can be achieved through steps S4021 to S4023:

[0090] Step S4021: Based on the target detection box corresponding to the target detection object, the target detection object in the detection image is cropped sequentially to obtain the cropped image and the image size corresponding to the cropped image.

[0091] Here, cropping the target detection object means cropping it according to the target detection box corresponding to the target detection object, so as to obtain the cropped image corresponding to the target detection box and the image size corresponding to the cropped image.

[0092] Step S4022: Extract features from the target detection object in the cropped image to obtain the location and category of the target detection object.

[0093] Step S4023: The detected image having the location and the category is determined as the feature image.

[0094] In this embodiment of the application, feature extraction may refer to extracting the position of the target detection object relative to the current vehicle and the category of the target detection object, such as truck, motorcycle or car.

[0095] In the embodiments of this application, any feature detection model can be used to extract features from the target object. After feature extraction, the position and category of the target object are marked on the detection box, and the image with position and category markings is determined as the feature image.

[0096] In some embodiments, target detection and feature detection of the data to be processed are performed on the GPU, target tracking is achieved based on the feature detection results, and data fusion is a comprehensive processing of the entire processing result. These two parts are completed on the CPU. Apart from these, the data processing methods provided in the embodiments of this application are all completed on the GPU, thereby accelerating data processing.

[0097] In this embodiment of the application, after obtaining the feature image, the feature image is copied from the GPU to the CPU, the image memory is returned, and the feature image is encapsulated and placed in the queue of the target tracking module.

[0098] Step S403: Based on the feature information, target tracking is performed on the target detection object in the feature image of the target tracking module to obtain target tracking information, and the tracking image with the target tracking information is sent to the data fusion module.

[0099] In some embodiments, the target tracking module retrieves feature images from the queue, performs target tracking on the target detection objects in the feature images, and continuously tracks the target detection objects in the feature images. If it is determined from the feature images that a target detection object is consistently present, the target tracking information of the vehicle is determined. Here, the target tracking information may be information such as the license plate and vehicle model of the target detection object. The image containing the target tracking information is determined as the tracking image.

[0100] In some embodiments, when the target tracking module finishes and there is no next-level processing submodule, the tracking image is encapsulated and sent to the queue of the data fusion module. When the target tracking module finishes and there is a next-level processing submodule, the next-level processing submodule processes the tracking image, encapsulates the processed tracking image, and sends it to the queue of the data fusion module.

[0101] Step S404: Perform data fusion on the target tracking information in the tracking image and the feature information of the target detection object in the data fusion module to obtain a target image with target detection object fusion information.

[0102] In some embodiments, data fusion of target tracking information and target detection object feature information in a tracking image refers to fusing the target tracking information obtained by the target tracking module and the target detection object feature information detected by the feature detection module, and then labeling them onto the image to obtain a target image with target detection object fusion information.

[0103] Step S405: Determine the target image with target detection object fusion information as the processing result.

[0104] In this embodiment of the application, after obtaining the processing result, the processing result can be returned to the decoding module according to the mapping relationship. The server determines whether to encode the inference information and send it to the current vehicle based on the target detection object fusion information on the target image.

[0105] In some embodiments, the decoding module is in the form of multiple instances, with a memory pool created for each instance, while the processing module is a global module with only one instance. Figure 5 This is a schematic diagram of the decoding module and processing module provided in the embodiments of this application, as shown below. Figure 5As shown, instances 1, 2, and 3 enter their corresponding decoding modules 1, 2, and 3. The decoded images are then sent to the processing modules for processing. Different execution environments in the processing modules process different instances, and the processing results are then returned to the corresponding decoding modules according to the mapping relationship.

[0106] This application embodiment performs video decoding and AI inference on the GPU, achieving a seamless connection between hardware decoding and AI inference. Performing AI inference on the GPU greatly reduces network inference latency, and through the management of the GPU memory pool, it effectively controls GPU resources and time consumption, thereby improving data processing efficiency.

[0107] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0108] This application's embodiment of the GPU-based end-to-end neural network inference acceleration method offloads time-consuming operations such as decoding, data preprocessing, network inference, and inference data post-processing to the GPU, and employs a pipelined approach to synchronize each stage, thereby accelerating the entire video data processing chain. The GPU-based end-to-end neural network inference acceleration method comprises two parts: a video decoding module and a processing module. The video decoding module is a multi-instance module, while the processing module is a global module.

[0109] In some embodiments, a video decoding instance corresponds to one video resource. Each instance registers with the processing module. The AI ​​constructs an independent execution environment for each instance and establishes a mapping relationship. The video decoding module sends the decoded data to the processing module, and the processing module returns the processed result image. Figure 6 This is a schematic diagram of the video decoding process provided in an embodiment of this application, such as... Figure 6 As shown, the video decoding process is implemented through steps S601 to S608:

[0110] Step S601: Resource initialization.

[0111] Here, resource initialization can involve initializing the video memory pool, registering the resource execution environment with the processing module, and establishing a mapping relationship between the environment and the processing module.

[0112] Step S602: Retrieve data from the queue.

[0113] In some embodiments, after acquiring video data, the video data is placed in a data decoding queue.

[0114] Step S603: Accelerate the data through hardware decoding.

[0115] After hardware decoding of the data, the decoded data will be obtained.

[0116] Step S604: Is there any free video memory in the video memory pool?

[0117] Here, when there is free video memory in the video memory pool, step S605 is executed; when there is no video memory in the video memory pool, step S602 is executed. When there is no video memory in the video memory pool, new data is acquired for decoding.

[0118] Step S605: Retrieve the decoded data from the video memory.

[0119] Step S606: Data encapsulation.

[0120] This refers to encapsulating the stored data.

[0121] Step S607: Task completed.

[0122] In some embodiments, after encapsulation, it is determined whether the task has ended. If so, step S608 is terminated; otherwise, step S602 is executed.

[0123] Step S608: Resource release.

[0124] Here, resource release refers to the processing module deregistering the resource execution environment, releasing the mapping relationship between the environment and the processing module, and sending the encapsulated data to the processing module's queue.

[0125] In some embodiments, the processing module maintains a resource execution environment and its internal mapping relationship for each resource. Data is transmitted to the processing module according to the mapping relationship, processed, and then transmitted to the decoding module according to the mapping relationship to determine whether the inferred data needs to be encoded and sent. The processing module also processes the data. Taking target detection and tracking in vehicle-to-everything (V2X) as an example, the processing module is divided into four parts: target detection, feature extraction, target tracking, and data fusion processing. All of these parts use a thread pool approach. Target tracking is based on feature detection results, and data fusion is a comprehensive processing of the entire AI processing results. These two parts are completed on the CPU. The rest of the processing flow is completed on the GPU to achieve acceleration. Figures 7 to 10 The diagrams shown illustrate the processing flow of target detection, feature extraction, target tracking, and data fusion. Furthermore, embodiments of this application can incorporate various AI processing functions into the overall processing flow as needed, based on business requirements.

[0126] Figure 7 This is a schematic diagram of the target detection process provided in the embodiments of this application, such as... Figure 7 As shown, the video decoding process is implemented through steps S701 to S709:

[0127] Step S701: Retrieve data from the queue.

[0128] Here, the data in the queue refers to the encapsulated data after hardware decoding.

[0129] Step S702: Traverse the data.

[0130] In some embodiments, traversing the data means traversing each frame of the image in the encapsulated data.

[0131] Step S703: Obtain video memory from the video memory pool.

[0132] Step S704: Preprocess the data.

[0133] Data preprocessing can include scaling, saving image data, and copying data to video memory.

[0134] Step S705: Traversal ends.

[0135] Here, when the image traversal ends, step S706 is executed; when the image traversal has not ended, step S702 is executed.

[0136] Step S706: Perform target detection inference on the data.

[0137] Step S707: Post-process the inference data.

[0138] Step S708: Return the hardware decoding memory.

[0139] Step S709: Data encapsulation, and the inference data is placed into the feature detection queue.

[0140] Figure 8 This is a schematic diagram of the feature extraction process provided in an embodiment of this application, such as... Figure 8 As shown, the feature extraction process is implemented through steps S801 to S809:

[0141] Step S801: Retrieve data from the feature detection queue.

[0142] Here, the data in the queue refers to the data after target detection.

[0143] Step S802: Traverse the data.

[0144] In some embodiments, traversing data refers to traversing each frame of the image in the encapsulated data.

[0145] Step S803: Obtain image data from the video memory pool.

[0146] Step S804: Crop the target and input it into the image data display memory.

[0147] The clipping target refers to the target object obtained from clipping target detection.

[0148] Step S805: Traversal ends.

[0149] Here, when the image traversal ends, step S806 is executed; when the image traversal has not ended, step S802 is executed.

[0150] Step S806: Perform feature detection and inference on the cropping target.

[0151] Step S807: Post-process the inference data.

[0152] Step S808: Copy the image data to the CPU and return the image data to the video memory.

[0153] Step S809: Data encapsulation, and inference data is placed into the target tracking queue.

[0154] Figure 9 This is a schematic diagram of the target tracking process provided in the embodiments of this application, such as... Figure 9 As shown, the target tracking process is implemented through steps S901 to S906:

[0155] Step S901: Retrieve data from the target tracking queue.

[0156] Step S902: Target tracking is achieved through the tracking module.

[0157] Step S903: Is there a next-layer network inference module?

[0158] Here, if there is a next-layer network inference module, execute step S904; otherwise, execute step S905.

[0159] Step S904: Data encapsulation, and the inference data is placed into the queue of the next layer network inference module.

[0160] Step S905: Return video memory.

[0161] Step S906: Data encapsulation, and inference data is placed into the data fusion queue.

[0162] Figure 10 This is a schematic diagram of the data fusion process provided in the embodiments of this application, such as... Figure 10 As shown, the data fusion process is implemented through steps S101 to S103:

[0163] Step S101: Retrieve data from the data fusion queue.

[0164] Step S102: Perform data fusion processing.

[0165] Step S103: Send the fused data to the decoding module according to the mapping relationship.

[0166] This application employs hardware decoding to accelerate data decoding, storing the decoded data in the GPU's video memory and performing data preprocessing for network inference on the GPU, thereby improving network runtime speed and reducing network inference latency. Secondly, it constructs a complete GPU-based processing flow encompassing data decoding, preprocessing, and network inference, accelerating video processing efficiency, reducing memory copying between the CPU and GPU, and saving bandwidth. Finally, it uses a memory pool method to manage GPU video memory, saving time overhead in memory allocation and deallocation, and effectively controlling GPU resource and time consumption.

[0167] Based on the above data processing methods Figure 11 This is a schematic diagram of the composition structure of the data processing device provided in the embodiments of this application, as shown below. Figure 11 As shown, the data processing device 110 includes an acquisition unit 111 and a processing unit 112. The acquisition unit 111 is used by the second processing unit to acquire first data from the first processing unit, preprocess the first data, and obtain data to be processed. The processing unit 112 is used to input the data to be processed into a processing module to obtain a processing result, which is then retrieved by the first processing unit. The processing module includes an algorithm model.

[0168] In some embodiments, if the processing module includes multiple processing sub-modules, the processing unit 112 is further configured to process the data to be processed in each of the processing sub-modules in a pipelined parallel manner.

[0169] In some embodiments, the preprocessing includes at least: data decoding; before preprocessing the first data, the apparatus further includes an initialization unit for initializing a video memory pool based on the first data.

[0170] In some embodiments, the first data is first video data, and the data to be processed obtained after the first video data is data decoding is the image to be processed; the initialization unit is further configured to determine the size of the video memory in the video memory pool according to the size of the first video data, and the size of the video memory corresponds to the size.

[0171] In some embodiments, the preprocessing further includes: scaling processing;

[0172] After obtaining the image to be processed, the device further includes a scaling unit for scaling the size of the image to be processed so that the scaled size meets the size requirements of the processing submodule to be input.

[0173] In some embodiments, the processing submodule includes a target detection module, a feature detection module, a target tracking module, and a data fusion module; the processing unit is further configured to sequentially input the image to be processed into the target detection module, the feature detection module, the target tracking module, and the data fusion module to obtain a processing result.

[0174] In some embodiments, the processing unit is further configured to: input the image to be processed into the target detection module; perform target detection on the image to be processed to obtain a detection image with a target object; and send the detection image to the feature detection module; extract target features from the detection image in the feature detection module to determine the feature information of the target object, obtain a feature image with the feature information, and send the feature image to the target tracking module; perform target tracking on the target object in the feature image in the target tracking module according to the feature information to obtain target tracking information, and send the tracked image with the target tracking information to the data fusion module; perform data fusion on the target tracking information in the tracked image in the data fusion module and the feature information of the target object to obtain a target image with target object fusion information; and determine the target image with target object fusion information as the processing result.

[0175] In some embodiments, the processing unit is further configured to perform target detection on the image to be processed, to obtain at least one target detection object and at least one detection box corresponding to each target detection object; to calculate the confidence score of at least one detection box corresponding to each target detection object using a non-maximum suppression algorithm, to obtain the confidence score value of each detection box; to determine the detection box with the highest confidence score value as the target detection box corresponding to each target detection object; and to determine the image to be processed containing the target detection box as the detection image containing the target detection object.

[0176] In some embodiments, the processing unit is further configured to: crop the target detection object in the detection image sequentially according to the target detection box corresponding to the target detection object to obtain a cropped image and an image size corresponding to the cropped image; extract features from the target detection object in the cropped image to obtain the position and category of the target detection object; and determine the detection image having the position and the category as the feature image.

[0177] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0178] It should be noted that, in the embodiments of this application, if the above-described data processing method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0179] This application provides a data processing device. Figure 12 This is a schematic diagram of the composition structure of the data processing device provided in the embodiments of this application, such as... Figure 12 As shown, the data processing device 120 includes at least a processor 121 and a computer-readable storage medium 122 configured to store executable instructions, wherein the processor 121 generally controls the overall operation of the data processing device. The computer-readable storage medium 122 is configured to store instructions and applications executable by the processor 121, and may also cache data to be processed or processed by various modules in the processor 121 and the data processing device 120, and may be implemented using flash memory or random access memory (RAM).

[0180] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the data processing method provided in this application, for example... Figure 2 The method shown.

[0181] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0182] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0183] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0184] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application. It should be understood that "an embodiment" or "one embodiment" mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence number of the above-described processes does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments of this application are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments.

[0185] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0186] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, the method comprising: The second processing unit obtains first data from the first processing unit, initializes the video memory pool according to the size of the first data, registers the resource execution environment with the processing module, and preprocesses the first data to obtain data to be processed; the preprocessing includes at least data decoding. The data to be processed is input into the processing module to obtain the processing result, which is then retrieved by the first processing unit. The processing module contains an algorithm model, creates an independent execution environment for each first data stream, and establishes a mapping relationship between the execution environment and the processing module. The processing module encapsulates the video memory and execution environment containing the data to be processed. After placing the encapsulated data into the queue of the processing module, the processing module cancels the resource execution environment and removes the mapping relationship between the execution environment and the processing module. Wherein, the first data is the first video data, and the initialization of the video memory pool based on the first data includes: The size of the video memory in the video memory pool is determined based on the size of the first video data, and the size of the video memory corresponds to the size.

2. The method according to claim 1, wherein if the processing module includes multiple processing sub-modules, the data to be processed is processed in each of the processing sub-modules in a pipelined parallel manner.

3. The method according to claim 2, wherein the data to be processed obtained after the first video data is data decoding is the image to be processed.

4. The method according to claim 3, wherein the preprocessing further comprises: Scaling; After obtaining the image to be processed, the method further includes: scaling the size of the image to be processed so that the scaled size meets the size requirements of the processing submodule to be input.

5. The method according to claim 4, wherein the processing submodule includes a target detection module, a feature detection module, a target tracking module, and a data fusion module; Accordingly, the data to be processed is input into the processing module to obtain the processing result, including: The image to be processed is sequentially input into the target detection module, the feature detection module, the target tracking module, and the data fusion module to obtain the processing result.

6. The method according to claim 5, wherein the step of sequentially inputting the image to be processed into the target detection module, the feature detection module, the target tracking module, and the data fusion module to obtain the processing result includes: The image to be processed is input into the target detection module, the target detection is performed on the image to be processed, a detection image with the target object is obtained, and the detection image is sent to the feature detection module; The target features are extracted from the detection image in the feature detection module to determine the feature information of the target detection object, thereby obtaining a feature image with feature information, and the feature image is sent to the target tracking module. Based on the feature information, target tracking is performed on the target detection object in the feature image of the target tracking module to obtain target tracking information, and the tracked image with the target tracking information is sent to the data fusion module. The target tracking information and the feature information of the target detection object in the tracking image in the data fusion module are fused to obtain a target image with target detection object fusion information. The target image containing the target detection object fusion information is determined as the processing result.

7. The method according to claim 6, wherein performing target detection on the image to be processed to obtain a detection image having a target object includes: Target detection is performed on the image to be processed to obtain at least one target detection object and at least one detection box corresponding to each target detection object; The confidence score of each detection box is calculated by using the non-maximum suppression algorithm. The detection box with the highest confidence score is determined as the target detection box corresponding to each target detection object. The image to be processed containing the target detection box is determined as a detection image containing the target object.

8. The method according to claim 7, wherein extracting target features from the detection image in the feature detection module to determine the feature information of the target detection object and obtain a feature image with feature information includes: Based on the target detection box corresponding to the target detection object, the target detection object in the detection image is cropped sequentially to obtain the cropped image and the image size corresponding to the cropped image; Feature extraction is performed on the target detection object in the cropped image to obtain the location and category of the target detection object; The detected image having the specified location and category is identified as the feature image.

9. A data processing apparatus, the apparatus comprising: The acquisition unit is used for the second processing unit to acquire first data from the first processing unit, initialize the video memory pool according to the size of the first data, register the resource execution environment with the processing module, preprocess the first data, and obtain data to be processed. The preprocessing includes at least data decoding; A processing unit is used to input the data to be processed into a processing module to obtain a processing result, which is then retrieved by the first processing unit. The processing module includes an algorithm model, creates an independent execution environment for each stream of first data, and establishes a mapping relationship between the execution environment and the processing module. It encapsulates the video memory and execution environment containing the data to be processed. After placing the encapsulated data into the queue of the processing module, the processing module unregisters the resource execution environment and releases the mapping relationship between the execution environment and the processing module. The first data is first video data. The acquisition unit is further configured to determine the size of the video memory in the video memory pool based on the size of the first video data, wherein the size of the video memory corresponds to the size.

Images