Image data processing method and apparatus, and electronic device

Abstract

Disclosed are an image data processing method and apparatus, and an electronic device. The image data processing method comprises: obtaining image data inputted from multiple input sources and received by a graphics processing unit (GPU); determining a target data size and a target data format of each piece of image data; on the basis of the target data size, the target data format, and the current image data, obtaining target image data of which a size is the target data size and a data format is the target data format; and copying the target image data to a designated memory of a central processing unit (CPU) by executing multiple threads in parallel, wherein each thread corresponds to a memory address of the CPU.

Description

Image data processing method, device and electronic equipment

[0001] Citation of Related Applications

[0002] This disclosure claims all rights and interests in the invention patent application with application number 202311680769.8, filed with the State Intellectual Property Office of the People's Republic of China on December 8, 2023, and entitled "Method, device and electronic device for processing image data", and incorporates the entire contents thereof into this disclosure by reference.

[0003] field

[0004] The present disclosure generally relates to the field of computer technology, and more particularly to a method, apparatus, and electronic device for processing image data.

[0005] background

[0006] With the increasing application of artificial intelligence technology, the demand for intelligent video analysis and processing is increasing. Among them, NVIDIA provides a relatively general intelligent video analysis and processing framework DeepStream. The DeepStream framework can decode videos, integrate video frames, infer and output the coordinate information of the algorithm results, such as the position and width and height data of the recognition results in the image; and in actual business usage scenarios, there is often a need to customize the integration of coordinate information and the original image. In this case, it is necessary to stably and efficiently obtain the image data of the original image from the GPU (graphics processing unit) and store the image data in the CPU (Central Processing Unit).

[0007] In the related art, there is a lack of a method for stably and efficiently acquiring image data from the GPU and storing it in the CPU memory while ensuring that the performance of the GPU and CPU are not affected.

[0008] Overview

[0009] In a first aspect, the present disclosure provides a method for processing image data, comprising:

[0010] Obtain image data input from multiple input sources received by the GPU;

[0011] determining a target data size and a target data format for each of the image data;

[0012] Based on the target data size, the target data format and the current image data, obtaining target image data having a size of the target data size and a data format of the target data format; and

[0013] The target image data is copied to a designated memory of the CPU by executing multiple threads in parallel, wherein each thread corresponds to a memory address of the CPU.

[0014] In some embodiments, obtaining target image data having a size of the target data size and a data format of the target data format based on the target data size, the target data format, and the current image data includes:

[0015] Determining whether the data size and data format of the current image data are the target data size and target data format;

[0016] If not, performing image data conversion on the image data to obtain target image data having a size equal to the target data size and a data format equal to the target data format;

[0017] If so, the image data is directly used as the target image data.

[0018] In certain embodiments, the copying of the target image data to a designated memory of a central processing unit (CPU) by executing multiple threads in parallel includes:

[0019] determining a target number of threads;

[0020] Determine the sub-target image data corresponding to each thread; and

[0021] The target number of threads are executed in parallel, and for each thread, the sub-target image data corresponding to the thread is copied to a designated memory of the CPU through a single instruction multiple data stream.

[0022] In some embodiments, the image data includes image data corresponding to multiple data sources, and determining the target number of threads includes:

[0023] determining the data source quantity of the data source included in the image data; and

[0024] determining the number of data sources as the target number of threads, wherein each data source corresponds to one thread; and

[0025] The determining of the sub-target image data corresponding to each thread includes:

[0026] The target image data corresponding to each data source is determined as sub-target image data corresponding to one thread.

[0027] In some embodiments, determining the target number of threads includes:

[0028] Obtaining the interval time for the GPU to receive new image data; and

[0029] Based on the interval time, a target number of the threads is determined.

[0030] In some embodiments, determining the target number of threads based on the interval time includes:

[0031] Obtain the historical copy time of target image data copied by different historical numbers of threads within a historical time period;

[0032] Determine the absolute value of the time difference between each of the historical copy times and the interval time; and

[0033] Determine the historical number of threads corresponding to the historical copy time with the smallest absolute value of the time difference as the target number of threads; and

[0034] The determining of the sub-target image data corresponding to each thread includes:

[0035] The target image data is evenly distributed to the target number of threads to obtain sub-target image data corresponding to each thread.

[0036] In some embodiments, determining the target number of threads based on the interval time includes:

[0037] determining the data volume of the target image data and determining the sub-data volume to be transmitted by one thread within the interval time; and

[0038] Dividing the data amount by the sub-data amount to obtain the target number of threads; and

[0039] The determining of the sub-target image data corresponding to each thread includes:

[0040] According to the amount of sub-data corresponding to each thread, the sub-target image data corresponding to each thread is determined.

[0041] In certain embodiments, the image data includes image data in multiple data formats, and determining the target number of threads includes:

[0042] determining multiple data formats included in the image data;

[0043] For each data format, determining a conversion time for converting the data format into the target data format;

[0044] Corresponding target image data corresponding to a data format with a conversion time greater than a preset time threshold to a separate thread; and

[0045] Assigning target image data corresponding to two or more data formats with conversion times less than or equal to a preset time threshold to a separate thread, and obtaining a target number of threads; and

[0046] The determining of the sub-target image data corresponding to each thread includes:

[0047] For each of the threads, the target image data in the data format corresponding to the thread is determined as the sub-target image data corresponding to the thread.

[0048] In a second aspect, the present disclosure provides an apparatus for processing image data, comprising:

[0049] An acquisition module configured to acquire image data inputted from multiple input sources received by a graphics processing unit (GPU);

[0050] a determination module configured to determine a target data size and a target data format of each of the image data;

[0051] a conversion module configured to obtain target image data having a size of the target data size and a data format of the target data format based on the target data size, the target data format and the current image data; and

[0052] The copy module is configured to copy the target image data to a designated memory of a central processing unit (CPU) by executing multiple threads in parallel, wherein each thread corresponds to a memory address of the CPU.

[0053] In certain embodiments, the conversion module is configured to:

[0054] Determining whether the data size and data format of the current image data are the target data size and target data format;

[0055] If not, performing image data conversion on the image data to obtain target image data having a size equal to the target data size and a data format equal to the target data format;

[0056] If so, the image data is directly used as the target image data.

[0057] In certain embodiments, the copy module comprises:

[0058] A first determining submodule is configured to determine a target number of threads;

[0059] A second determining submodule is configured to determine the sub-target image data corresponding to each thread; and

[0060] The execution submodule is configured to execute the target number of threads in parallel, and copy the sub-target image data corresponding to each thread to the designated memory of the CPU through a single instruction multiple data stream.

[0061] In certain embodiments, the image data includes image data corresponding to multiple data sources, and the first determining submodule is configured to:

[0062] determining the data source quantity of the data source included in the image data; and

[0063] determining the number of data sources as the target number of threads, wherein each data source corresponds to one thread; and

[0064] As the second determination submodule, it is configured as follows:

[0065] The target image data corresponding to each data source is determined as sub-target image data corresponding to one thread.

[0066] In certain embodiments, the first determining submodule comprises:

[0067] an acquiring unit, configured to acquire an interval time for the GPU to receive new image data;

[0068] The determining unit is configured to determine the target number of the threads based on the interval time.

[0069] In certain embodiments, the determining unit is configured to:

[0070] Obtain the historical copy time of target image data copied by different historical numbers of threads within a historical time period;

[0071] Determine the absolute value of the time difference between each of the historical copy times and the interval time; and

[0072] Determine the historical number of threads corresponding to the historical copy time with the smallest absolute value of the time difference as the target number of threads; and

[0073] The second determining submodule is configured as follows:

[0074] The target image data is evenly distributed to the target number of threads to obtain sub-target image data corresponding to each thread.

[0075] In certain embodiments, the determining unit is configured to:

[0076] determining the data volume of the target image data and determining the sub-data volume to be transmitted by one thread within the interval time; and

[0077] Dividing the data amount by the sub-data amount to obtain the target number of threads; and

[0078] The second determining submodule is configured as follows:

[0079] According to the amount of sub-data corresponding to each thread, the sub-target image data corresponding to each thread is determined.

[0080] In certain embodiments, the image data includes image data in multiple data formats, and the first determining submodule is configured to:

[0081] determining multiple data formats included in the image data;

[0082] For each data format, determining a conversion time for converting the data format into the target data format;

[0083] Corresponding target image data corresponding to a data format with a conversion time greater than a preset time threshold to a separate thread; and

[0084] Assigning target image data corresponding to two or more data formats with conversion times less than or equal to a preset time threshold to a separate thread, and obtaining a target number of threads; and

[0085] The second determining submodule is configured as follows:

[0086] For each of the threads, the target image data in the data format corresponding to the thread is determined as the sub-target image data corresponding to the thread.

[0087] In a third aspect, the present disclosure provides an electronic device comprising: a processor and a memory, wherein the processor is configured to execute a processing program for image data stored in the memory to implement the image data processing method described in the present disclosure.

[0088] In a fourth aspect, the present disclosure provides a storage medium, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the image data processing method described in the present disclosure.

[0089] In certain embodiments, in order to improve the efficiency of copying image data from the GPU to the CPU, the various image data are unified into a target data size and a target data format. This can avoid problems such as memory leaks and program crashes caused by image data of different data formats after the image data in the graphics processor is transferred to the central processing unit. Since the data format of the image data is unified, the image data can be easily copied to the central processing unit in a multi-threaded manner, thereby speeding up the copy efficiency of the image data. This achieves the goal of stably copying image data from the GPU to the CPU while improving the image data copy efficiency from the GPU to the CPU.

[0090] BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0092] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0093] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0094] FIG1 is a schematic diagram of a scenario of multiple input sources of a graphics processor provided by an embodiment of the present disclosure;

[0095] FIG2 is a flow chart of an embodiment of a method for processing image data provided by an embodiment of the present disclosure;

[0096] FIG3 is a flow chart of another method for processing image data according to an embodiment of the present disclosure;

[0097] FIG4 is a flow chart of another method for processing image data according to an embodiment of the present disclosure;

[0098] FIG5 is a block diagram of an embodiment of an apparatus for processing image data provided by an embodiment of the present disclosure; and

[0099] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

[0100] Details

[0101] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0102] The disclosure below provides many different embodiments or examples for implementing different structures of the present disclosure. To simplify the present disclosure, the components and settings of specific examples are described below. Of course, these are merely examples and are not intended to limit the present disclosure. In addition, the present disclosure may repeat reference numbers and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0103] GPU is a microprocessor used to process image and graphics related operations. In the field of edge computing, GPU can obtain image or video data from multiple different external input sources, such as multiple cameras or multiple MP4 files. With the increasing application of artificial intelligence technology, the need for analysis and processing of intelligent videos is increasing. In GPU image processing, the output is generally coordinate data, and in actual business usage scenarios, there is often a need to customize the integration of coordinate information and original images, such as drawing customized information, saving it locally or in a cache and then transmitting it. However, no effective data processing method has been proposed in the relevant technology, which can stably and efficiently copy the image data of the GPU to the CPU for processing. Based on this, the present disclosure provides a method, device and electronic device for processing image data, which can achieve stable copying of image data from the GPU to the CPU while improving the efficiency of copying image data from the GPU to the CPU.

[0104] To facilitate understanding of the image data processing method provided by the present disclosure, the following first illustrates a schematic diagram of a scenario with multiple input sources of a GPU involved in the method:

[0105] 1 is a schematic diagram of a multi-channel input source scenario for a graphics processor according to an embodiment of the present disclosure. As shown in FIG1 , the multi-channel input source scenario for the GPU may include: a graphics processor 10, a camera 11, a camera 12, a video camera 13, and a camera 14.

[0106] The graphics processor 10 is a microprocessor specifically used for performing image and graphics related operations on a terminal. The terminal may include but is not limited to personal computers, workstations, game consoles, and some mobile devices (such as tablet computers, smart phones, etc.).

[0107] The cameras 11, 12, and 14 can be used as video input devices for video conferencing, telemedicine, and real-time monitoring in daily life, and can record audio and video. As for the specific type and model of the cameras, the embodiments of the present disclosure do not limit this.

[0108] The camera 13 is a device used for shooting images in daily life. The video files shot by the camera 13 may generally be MP4 files or other video files, which is not limited in the embodiment of the present disclosure.

[0109] In certain embodiments, after capturing corresponding video images, the aforementioned cameras 11, 12, 13, and 14 may generate corresponding image data and send the image data to the GPU for image and graphics-related operations. It should be noted that the aforementioned cameras 11, 12, 13, and 14 are merely examples of multiple input sources for the graphics processor 10. In actual applications, the input sources of the graphics processor 10 are not limited to the aforementioned cameras 11, 12, 13, and 14.

[0110] With the increasing application of artificial intelligence technology, the need for intelligent video analysis and processing is growing. In GPU image processing, the output is generally coordinate data. However, in actual business usage scenarios, there is often a need to customize the integration of coordinate information and original images, such as drawing customized information, saving it locally or in a cache for transmission, etc. This requires copying the image data received by the GPU from multiple data sources to the CPU.

[0111] However, the related art has not proposed an effective data processing method that can stably and efficiently copy image data from the GPU to the CPU for processing. Based on this, the present disclosure provides an image data processing method, device, and electronic device that can stably copy image data from the GPU to the CPU while improving the efficiency of the GPU-to-CPU image data copy.

[0112] The image data processing method provided by the present disclosure is further explained below with reference to specific embodiments in conjunction with the accompanying drawings. The embodiments do not constitute a limitation of the embodiments of the present disclosure.

[0113] Refer to Figure 2, which is a flow chart of an embodiment of a method for processing image data provided by an embodiment of the present disclosure. As shown in Figure 2, the process may include the following steps:

[0114] Step 201: Obtain image data inputted from multiple input sources received by a graphics processor;

[0115] Step 202: Determine the target data size and target data format of each image data;

[0116] Step 203: Based on the target data size, the target data format, and the current image data, obtain target image data having a size equal to the target data size and a data format equal to the target data format; and

[0117] Step 204 : Copy the target image data to a designated memory of the central processing unit by executing multiple threads in parallel, wherein each thread corresponds to a memory address of the central processing unit.

[0118] The graphics processor (GPU), also known as a display core, visual processor, or display chip, is a microprocessor specifically designed to perform image and graphics operations on personal computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). It performs image-related operations and saves the image data.

[0119] The above-mentioned image data refers to the relevant data that constitutes the image, which may be image data corresponding to any data format. The above-mentioned data format refers to the image data file format, which may include but is not limited to: RGBA (Red, Green, Blue and Alpha, color space), RGB (Red, Green, Blue), HSV (Hue, Saturation, Value), and YUV (color encoding method).

[0120] In certain embodiments, for different application scenarios, a GPU can simultaneously receive input from multiple data sources, which may correspond to different cameras or other image acquisition devices. Based on this, the execution subject of the embodiments of the present disclosure can obtain image data received by the graphics processor, where the image data may include image data in one or more different data formats.

[0121] In certain embodiments, after receiving image data, the graphics processor stores the received image data in a preset storage area. To this end, the execution subject of the embodiments of the present disclosure may first determine a preset storage area in the graphics processor configured to store image data, and then obtain image data currently stored in the preset storage area, including one or more different data formats.

[0122] In certain embodiments, the execution subject of the embodiment of the present disclosure may directly obtain the image data received by the graphics processor after the graphics processor receives image data input from multiple input sources.

[0123] It should be noted that the executor of the embodiment of the present disclosure may be a GPU. Here, the image data currently stored in the GPU and containing one or more different data formats are obtained. The image data stored in the GPU is not obtained into other storage media, but is only processed in the next step.

[0124] In some embodiments, the target data size refers to the image data size after the image data is converted.

[0125] In some embodiments, the target data format refers to the data format to which the image data is to be converted, which may be RGBA format or other formats, and the present disclosure does not limit this. It should be noted that the target data format here is a data format.

[0126] In certain embodiments, for different application scenarios, the GPU can simultaneously receive input from multiple data sources. The multiple data sources can correspond to different cameras or other image acquisition devices. For the output of image data for different scenarios, the output of the multiple data sources can have different data formats and sizes. In certain embodiments, for a specific location, the processed image data outputs of cameras at locations of different importance have different image sizes. Some cameras have a target data size of 1080P, and some cameras have a target data size of 720P. In this scenario, it is necessary to set the target data size of which image data is 1080P and which is 720P; or in the case where the quality requirements for the image data output are not high, the target data size of the image data can also be uniformly set to 720P. Therefore, in the present disclosure, after obtaining the image data input by the multiple input sources received by the GPU, it is necessary to further determine the target data size and target data format of each image data.

[0127] In certain embodiments, the execution entity of the disclosed embodiments is connected to a visual interface, on which a user selects or enters a target data size and a target data format. Based on this, the execution entity of the disclosed embodiments can obtain the target data size and target data format of the image data to be converted input by the user through the visual interface.

[0128] In certain embodiments, when copying image data within the GPU to the designated memory of the CPU, image data in different data formats may cause memory leaks, program crashes, and other problems. Therefore, in order to improve the efficiency of copying image data from the GPU to the CPU, the executor of the embodiment of the present disclosure may uniformly convert image data of different data formats input by multiple input sources received by the GPU into target image data of a target data size and a target data format before copying the image data within the GPU to the CPU.

[0129] In some embodiments, the image data received by the GPU may include image data having a size of the target data size and a data format of the target data format. Therefore, before performing format and size conversion on the image data received by the GPU, such image data having a size of the target data size and a data format of the target data format may be excluded.

[0130] In some embodiments, it may be determined whether the data size and data format of the current image data are the target data size and target data format.

[0131] In some embodiments, if the data size and data format of the current image data are the target data size and target data format, the image data can be directly used as the target image data.

[0132] On the contrary, if the data size and data format of the current image data are not the above target data size or target data format, the above image data can be converted to obtain target image data with the target data size and the target data format.

[0133] In certain embodiments, the execution subject of the embodiment of the present disclosure may perform data format conversion on image data in different data formats by calling an image format conversion interface.

[0134] In certain embodiments, the execution subject of the embodiment of the present disclosure may obtain the target data size and target data format of the target image data, as well as the computing unit required to perform format conversion on the image data of each data format.

[0135] Based on this, the above-mentioned current image data, target data size, target data format of the target image data, and the calculation unit required for format conversion of image data of each data format can be input into the called image format conversion interface to obtain the target image data output by the image format conversion interface, whose size is the target data size and data format is the target data format.

[0136] In certain embodiments, when image data is format converted through an image format conversion interface, the following operations may be performed within a determined computing unit: first, image information of each image data may be obtained. The image information here refers to attribute information of the image data, which may include but is not limited to information such as the image size and the image data size.

[0137] Afterwards, the data format of the image data can be determined based on the image information. Since the computing unit cannot determine the data format corresponding to each image data after acquiring the image data, the image data can be converted based on the data format of the image data by acquiring the image information of each image data and determining the data format of each image data based on the image information.

[0138] Since image data of different data formats require different methods for format conversion, format conversion formulas for converting image data of different formats can be pre-stored. After determining the data format of the image data, a target format conversion formula corresponding to the data format can be determined from a plurality of pre-set format conversion formulas. The image data may include image data of multiple data formats. Therefore, when determining the target format conversion formula, a target format conversion formula corresponding to each data format of the image data can be determined separately.

[0139] Finally, the target data size and target data format can be used to convert the image data using the target format conversion formula to obtain target image data having the target data format. When converting the image data using the target format conversion formula, the target data size can be used as a target to convert the image data to obtain target image data having the target data size.

[0140] In certain embodiments, in order to further accelerate the format conversion of image data, the execution subject of the embodiment of the present disclosure may call multiple threads to simultaneously execute the conversion of the data format of the image data into the target data format, thereby converting the image data of various different data formats input by multiple input sources received by the GPU into target image data of the target data format, wherein the image data of each data source in the image data can be format converted by one thread.

[0141] In some embodiments, the memory address may be any memory address in the central processing unit for storing image data.

[0142] In certain embodiments, since the GPU receives image data input from multiple data sources in batches, after receiving a batch of image data, the obtained image data can be stored in a preset storage area, and after receiving the next batch of image data, the obtained new image data can be stored in the same storage area, which will result in overwriting the previously acquired image data. Therefore, in the embodiment of the present disclosure, it is necessary to quickly copy the converted target image data to avoid the loss of the obtained image data.

[0143] Based on this, in order to speed up the transmission efficiency of the target image data, the execution subject of the embodiment of the present disclosure can transfer the target image data to the designated memory of the central processing unit by executing multiple threads in parallel, and correspond each thread to a memory address of the central processing unit. Based on this, the target image data of the GPU can be quickly copied to the central processing unit.

[0144] In certain embodiments, in order to further speed up the transmission efficiency of the target image data, the execution body of the embodiment of the present disclosure uses SIMD (Single Instruction Multiple Data) to transfer the target image data corresponding to each thread to the designated memory of the central processing unit for storage, so that the CPU can save the data locally or transmit it to the server through a network protocol.

[0145] As for how to execute multiple threads in parallel to transfer the target image data to the central processing unit, it can be explained below through the process shown in Figure 3, which will not be described in detail here.

[0146] The technical solution provided by the embodiment of the present disclosure obtains image data input from multiple input sources received by the graphics processor, determines the target data size and target data format of each image data, obtains target image data of the target data size and the target data format based on the target data size and the target data format and the current image data, and copies the target image data to the designated memory of the central processing unit (CPU) by executing multiple threads in parallel, wherein each thread corresponds to a memory address of the CPU. This technical solution, by unifying each channel of image data into the target data size and target data format in order to improve the efficiency of copying image data from the GPU to the CPU, can avoid problems such as memory leaks and program crashes caused by image data of different data formats after transferring the image data from the graphics processor to the central processing unit. Since the data format of the image data is unified, the image data can be transferred to the central processing unit through multiple threads, thereby accelerating the transfer efficiency of the image data, achieving stable copying of image data from the GPU to the CPU while improving the efficiency of copying image data from the GPU to the CPU.

[0147] See Figure 3, which is a flowchart of another embodiment of a method for processing image data provided by one embodiment of the present disclosure. The process shown in Figure 3, based on the process shown in Figure 2, specifically describes how to execute multiple threads in parallel to transfer target image data to a central processing unit. As shown in Figure 3, the process may include the following steps:

[0148] Step 301: Determine the target number of threads;

[0149] Step 302: Determine the sub-target image data corresponding to each thread; and

[0150] Step 303 : executing the target number of threads in parallel, and copying the sub-target image data corresponding to each thread to the designated memory of the central processing unit through a single instruction multiple data flow.

[0151] The following is a unified description of step 301 and step 302:

[0152] In certain embodiments, in order to increase the transfer speed of target image data while avoiding waste of resources, the execution entity of the disclosed embodiment may determine a target number of threads and the sub-target image data corresponding to each thread, thereby executing the target number of threads in parallel according to the sub-target image data corresponding to each thread to transfer the target image data stored in the graphics processor to the central processing unit.

[0153] In some embodiments, a target storage area configured to store target image data in the central processing unit may be determined, and the target image data in the graphics processor may be stored in the target storage area in the central processing unit.

[0154] In some embodiments, the above-mentioned image data may include image data corresponding to multiple data sources, where the data source refers to the input source of the image data, such as video data in different data formats input by multiple different cameras, and the video data may include image data in multiple different data formats.

[0155] Based on this, the execution subject of the embodiment of the present disclosure may determine the number of data sources included in the image data, correspond each data source to a thread, and thus determine the number of data sources as the target number of threads.

[0156] Afterwards, the target image data corresponding to each data source may be determined as sub-target image data of a thread.

[0157] In certain implementations, the execution subject of the disclosed embodiments may determine the target number of threads according to the type of data format included in the image data.

[0158] Based on this, the execution subject of the embodiment of the present disclosure can determine the number of types of data formats included in the image data in the GPU, and correspond each type of data format to a thread, thereby determining the above number of types as the target number of threads.

[0159] Afterwards, the target image data corresponding to each data format may be determined as sub-target image data corresponding to one thread.

[0160] In some embodiments, the interval time for the graphics processor to receive new image data can be obtained, and the target number of threads can be determined based on the interval time, thereby avoiding the situation where the original image data is not read in time and is lost due to the new image data overwriting the original image data after the graphics processor receives the new image data.

[0161] In certain embodiments, the execution subject of the disclosed embodiments may obtain historical copy times for target image data by different numbers of threads within a historical time period. In certain embodiments, within a historical time period, the historical transfer time for three threads executing in parallel to copy the same amount of target image data is 30 seconds, and the historical transfer time for four threads executing in parallel to copy the same amount of target image data is 20 seconds.

[0162] Based on this, the absolute value of the time difference between each historical copy time and the interval time can be determined, and the historical number of threads corresponding to the historical transfer time with the smallest absolute value of the time difference can be determined as the target number of threads. For example, if the interval time is 25 seconds, and during the historical time period, it takes 30 seconds for three threads to copy the target image data in parallel, and 20 seconds for four threads to copy the same amount of target image data in parallel, then the target number of threads can be determined as 4.

[0163] Afterwards, in some implementations, the target image data may be evenly distributed to a target number of threads according to an even distribution principle, to obtain sub-target image data corresponding to each thread.

[0164] In some embodiments, the target image data may be distributed to the target number of threads according to the data source of the target image data, thereby obtaining sub-target image data corresponding to each thread.

[0165] In some embodiments, the target number of threads may be determined directly based on the data volume of the target image data.

[0166] In some embodiments, the amount of target image data can be determined, and the amount of sub-data to be transmitted by a thread within an interval can be determined. This amount of data can then be divided by the amount of sub-data to obtain the target number of threads. The sub-target image data corresponding to each thread can then be determined based on the amount of sub-data corresponding to each thread.

[0167] In some embodiments, assuming that the interval time is 30 seconds, and continuing to assume that the amount of sub-data transmitted by each thread within 30 seconds is 10M, and the amount of target image data is 50M, then dividing the data amount by the above sub-data amount, the target number of threads can be obtained as 5, and the target image data is divided according to each sub-data amount of 10M to obtain the sub-target image data corresponding to each thread.

[0168] In some embodiments, the target number of threads may be determined based on the conversion time of converting different data formats into the target data format.

[0169] In certain embodiments, multiple data formats corresponding to the image data may be determined, and for each data format, a conversion time for converting the data format into a target data format may be determined.

[0170] Based on this, the target image data corresponding to the data format with a conversion time greater than the preset time threshold can be corresponded to a separate thread, and the target image data corresponding to two or more data formats with a conversion time less than or equal to the preset time threshold can be corresponded to a separate thread, thereby obtaining the target number of threads.

[0171] Afterwards, for each thread, the target image data in the data format corresponding to the thread may be determined as the sub-target image data corresponding to the thread.

[0172] In certain embodiments, it is assumed that the image data includes three data formats: a first data format, a second data format, and a third data format. Continuing to assume that the conversion time corresponding to the conversion from the first data format to the target data format is 15 seconds, the conversion time corresponding to the conversion from the second data format to the target data format is 25 seconds, and the conversion time corresponding to the conversion from the third data format to the target data format is 50 seconds, and continuing to assume that the time threshold is 28 seconds, then the conversion time of the third data format is longer and can be corresponded to a separate thread, while the conversion time of the first data format and the second data format is shorter, and these two data formats can be corresponded to one thread. Based on this, the target number of threads can be obtained as 2, the sub-target image data corresponding to the first thread is the target image data corresponding to the third data format, and the sub-target image data corresponding to the second thread is the target image data corresponding to the first data format and the second data format.

[0173] In some embodiments, after determining the target number of threads, the target number of threads can be executed in parallel, thereby transferring the sub-target image data corresponding to each thread to the execution memory of the central processing unit, thereby quickly transferring the target image data in the GPU to the central processing unit.

[0174] In some embodiments, for each target number of threads, the sub-target image data corresponding to the thread can be transferred to the designated memory of the central processing unit through a single instruction multiple data stream, thereby achieving further rapid transfer of the sub-target image data corresponding to each thread to the designated memory of the central processing unit.

[0175] The technical solution provided by the embodiments of the present disclosure determines a target number of threads, determines the sub-target image data corresponding to each thread, executes the target number of threads in parallel, and copies the sub-target image data corresponding to each thread to the designated memory of the central processing unit through a single instruction multiple data stream. This technical solution, by determining the target number of threads and the sub-target image data corresponding to each thread, executes the target number of threads in parallel according to the sub-target image data corresponding to each thread, thereby improving the copying speed of the target image data while avoiding resource waste, thereby achieving stable copying of image data from the GPU to the CPU and improving the efficiency of image data copying from the GPU to the CPU.

[0176] Referring to FIG4 , which is a flow chart of another embodiment of a method for processing image data provided by an embodiment of the present disclosure, the process may include the following:

[0177] First, the image data on the GPU can be converted into a data format. All image data output by the previous process, regardless of the format, can be converted into RGBA format.

[0178] In some embodiments, parameters such as the image data size and the computational units required for the conversion can be set. The conversion can then begin. After the conversion is complete, the data can be directly read, but its address will be overwritten by the next batch of data, making it impossible to read the data. Therefore, the data must be quickly copied to a new memory address.

[0179] Because a single batch of data can involve multiple input sources, a multi-threaded approach is designed to process the data. Simultaneously, SIMD acceleration technology is used within each thread, accelerating data copying to approximately 20ms at 1080p. SIMD acceleration is achieved through NEON, a SIMD extension architecture for ARM Cortex-A and Cortex-R series processors.

[0180] The technical solution provided by an embodiment of the present disclosure provides a stable, reliable and efficient image acquisition solution. It can not only stably obtain the required image data from the GPU data stream, but also, because the acquired data specifications are unified, multi-threaded NEON acceleration can be used for memory transfer, further improving data transmission efficiency and reducing pipeline blocking time.

[0181] Refer to Figure 5, which is a block diagram of an embodiment of an image data processing device provided by an embodiment of the present disclosure. As shown in Figure 5, the device may include:

[0182] An acquisition module 51 is configured to acquire image data inputted from multiple input sources received by a graphics processing unit (GPU);

[0183] a determination module 52 configured to determine a target data size and a target data format of each of the image data;

[0184] a conversion module 53 configured to obtain target image data having a size of the target data size and a data format of the target data format based on the target data size, the target data format and the current image data; and

[0185] The copy module 54 is configured to copy the target image data to a designated memory of a central processing unit (CPU) by executing multiple threads in parallel, wherein each thread corresponds to a memory address of the CPU.

[0186] In certain embodiments, the conversion module 53 is configured to:

[0187] Determining whether the data size and data format of the current image data are the target data size and target data format;

[0188] If not, performing image data conversion on the image data to obtain target image data having a size equal to the target data size and a data format equal to the target data format;

[0189] If so, the image data is directly used as the target image data.

[0190] In some embodiments, the copy module 54 includes:

[0191] A first determining submodule is configured to determine a target number of threads;

[0192] A second determining submodule is configured to determine the sub-target image data corresponding to each thread; and

[0193] The execution submodule is configured to execute the target number of threads in parallel, and for each thread, copy the sub-target image data corresponding to the thread to the designated memory of the CPU through a single instruction multiple data stream.

[0194] In certain embodiments, the image data includes image data corresponding to multiple data sources, and the first determining submodule is configured to:

[0195] determining the data source quantity of the data source included in the image data; and

[0196] determining the number of data sources as the target number of threads, wherein each data source corresponds to one thread; and

[0197] As the second determination submodule, it is configured as follows:

[0198] The target image data corresponding to each data source is determined as sub-target image data corresponding to one thread.

[0199] In certain embodiments, the first determining submodule comprises:

[0200] an acquiring unit configured to acquire an interval time for the GPU to receive new image data; and

[0201] The determining unit is configured to determine the target number of the threads based on the interval time.

[0202] In certain embodiments, the determining unit is configured to:

[0203] Obtain the historical copy time of target image data copied by different historical numbers of threads within a historical time period;

[0204] Determine the absolute value of the time difference between each of the historical copy times and the interval time; and

[0205] Determine the historical number of threads corresponding to the historical copy time with the smallest absolute value of the time difference as the target number of threads; and

[0206] The second determining submodule is configured as follows:

[0207] The target image data is evenly distributed to the target number of threads to obtain sub-target image data corresponding to each thread.

[0208] In certain embodiments, the determining unit is configured to:

[0209] determining the data volume of the target image data and determining the sub-data volume to be transmitted by one thread within the interval time; and

[0210] Dividing the data amount by the sub-data amount to obtain the target number of threads; and

[0211] The second determining submodule is configured as follows:

[0212] According to the amount of sub-data corresponding to each thread, the sub-target image data corresponding to each thread is determined.

[0213] In certain embodiments, the image data includes image data in multiple data formats, and the first determining submodule is configured to:

[0214] determining multiple data formats included in the image data;

[0215] For each data format, determining a conversion time for converting the data format into the target data format;

[0216] Corresponding target image data corresponding to a data format with a conversion time greater than a preset time threshold to a separate thread; and

[0217] Assigning target image data corresponding to two or more data formats with conversion times less than or equal to a preset time threshold to a separate thread, and obtaining a target number of threads; and

[0218] The second determining submodule is configured as follows:

[0219] For each of the threads, the target image data in the data format corresponding to the thread is determined as the sub-target image data corresponding to the thread.

[0220] As shown in FIG6 , a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure includes a processor 61, a communication interface 62, a memory 63, and a communication bus 64. The processor 61, the communication interface 62, and the memory 63 communicate with each other via the communication bus 64.

[0221] a memory 63 configured to store computer programs;

[0222] The processor 61 is configured to execute the program stored in the memory 63 to implement the image data processing method described in the present disclosure, including:

[0223] Obtain image data input from multiple input sources received by a graphics processing unit (GPU);

[0224] determining a target data size and a target data format for each of the image data;

[0225] Based on the target data size, the target data format and the current image data, obtaining target image data having a size of the target data size and a data format of the target data format; and

[0226] The target image data is copied to a designated memory of a central processing unit (CPU) by executing multiple threads in parallel, wherein each thread corresponds to a memory address of the CPU.

[0227] An embodiment of the present disclosure further provides a storage medium having a computer program stored thereon, which implements the image data processing method described in the present disclosure when the computer program is executed by a processor.

[0228] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0229] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course, by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiment.

[0230] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0231] The foregoing description is intended only to provide specific embodiments of the present disclosure, which will enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments shown herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing image data, which includes: obtaining image data input by multiple input sources received by a graphics processing unit (GPU); determining a target data size and a target data format for each of the image data; obtaining target image data with a size of the target data size and a data format of the target data format based on the target data size, the target data format, and the current image data; and copying the target image data to a specified memory of a central processing unit (CPU) by parallelly executing multiple threads, wherein each thread corresponds to a memory address of the CPU.

2. The method according to claim 1, wherein the obtaining the target image data with a size of the target data size and a data format of the target data format based on the target data size, the target data format, and the current image data, includes: judging whether a data size and a data format of the current image data are the target data size and the target data format; if not, performing image data conversion on the image data to obtain target image data with a size of the target data size and a data format of the target data format; if so, directly using the image data as the target image data.

3. The method according to claim 1 or 2, wherein the copying the target image data to a specified memory of a central processing unit (CPU) by parallelly executing multiple threads, includes: determining a target number of the threads; determining sub-target image data corresponding to each thread; and parallelly executing the target number of the threads, and for each thread, copying the sub-target image data corresponding to the thread to the specified memory of the CPU in a single instruction multiple data stream manner.

4. The method according to claim 3, wherein the image data includes image data corresponding to multiple data sources, and the determining the target number of the threads, includes: determining a number of data sources of the data sources included in the image data; determining the number of data sources as the target number of the threads, wherein each data source corresponds to one thread; and the determining the sub-target image data corresponding to each thread includes: determining the target image data corresponding to each data source as the sub-target image data corresponding to one thread.

5. The method according to claim 3 or 4, wherein the determining the target number of the threads, includes: obtaining an interval time for the GPU to receive new image data; and determining the target number of the threads based on the interval time.

6. The method according to claim 5, wherein the determining the target number of the threads based on the interval time, includes: obtaining historical copy times for different historical numbers of threads to copy target image data within a historical time period; determining an absolute value of a time difference between each historical copy time and the interval time; and determining the historical number of the thread corresponding to the historical copy time with the smallest absolute value of the time difference as the target number of the threads; and the determining the sub-target image data corresponding to each thread includes: Evenly distribute the target image data to the target number of threads to obtain sub-target image data corresponding to each thread.

7. The method according to claim 5, wherein determining the target number of threads based on the interval time comprises: determining the data volume of the target image data and determining the sub-data volume transmitted by one thread within the interval time; and dividing the data volume by the sub-data volume to obtain the target number of threads; and the determining of the sub-target image data corresponding to each thread comprises: determining the sub-target image data corresponding to each thread according to the sub-data volume corresponding to each thread.

8. The method according to any one of claims 3 to 7, wherein the image data includes image data in multiple data formats, and determining the target number of threads comprises: determining the multiple data formats included in the image data; for each data format, determining the conversion time for converting the data format to the target data format; corresponding the target image data corresponding to the data format with a conversion time greater than the preset time threshold to a separate thread; and corresponding the target image data corresponding to two or more data formats with a conversion time less than or equal to the preset time threshold to a separate thread to obtain the target number of threads; and the determining of the sub-target image data corresponding to each thread comprises: for each thread, determining the target image data in the data format corresponding to the thread as the sub-target image data corresponding to the thread.

9. An apparatus for processing image data, which comprises: an acquisition module configured to acquire image data input by a plurality of input sources received by a graphics processing unit (GPU); a determination module configured to determine the target data size and the target data format of each of the image data; a conversion module configured to obtain target image data with a size of the target data size and a data format of the target data format based on the target data size, the target data format, and the current image data; and a copy module configured to copy the target image data to a specified memory of a central processing unit (CPU) by parallel execution of multiple threads, wherein each thread corresponds to a memory address of the CPU.

10. An electronic device, which comprises: a processor and a memory, the processor being configured to execute an image data processing program stored in the memory to implement the image data processing method according to any one of claims 1 to 8.

Images