Image data processing method and device, readable storage medium and electronic equipment

Abstract

Embodiments of the present disclosure disclose a kind of image data processing method, device, computer readable storage medium and electronic equipment, wherein the method comprises: determining the processing condition information of the image data processing unit of the target image data processing unit cascade next stage;In response to based on processing condition information, determine that image data processing progress meets the start condition of the image data processing stage that next stage image data processing unit will carry out, control next stage image data processing unit with the image data output by target image data processing unit carries out the image data processing of next image data processing stage.This embodiment of the present disclosure realizes that each level image data processing unit processes image data in the way of pipeline, greatly reduces the time that each level image data processing unit waits for the image data output by previous stage image data processing unit, thereby reduces the delay of image data processing, improves the speed of image data processing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, and particularly relates to an image data processing method and device, a computer readable storage medium and an electronic device. BACKGROUND

[0002] With the popularity of intelligent electronic devices, the application of image processing technology is more and more widely. The image processing technology can include various technical sub-classes such as image classification, target detection, and character recognition. In some technical fields, there is a high requirement for the real-time performance of image processing. For example, in the field of automatic driving, it is necessary to use the captured image for target detection in real time, so as to control the driving of the vehicle.

[0003] Generally, from the acquisition of the original image data from the camera to the final output of the image processing result, it is necessary to pass through multiple types of image data processing links. The current image data processing method usually needs to output the image data to the next link for processing after the image data processing in the previous link is completed. In the case of a large number of image data processing links, the delay of image data processing is large. SUMMARY

[0004] Embodiments of the present disclosure provide an image data processing method and device, a computer readable storage medium and an electronic device.

[0005] Embodiments of the present disclosure provide an image data processing method, which comprises: determining processing condition information of a subsequent image data processing unit cascaded with a target image data processing unit, wherein the processing condition information is used to represent the processing condition of the image data processing stage to be performed by the subsequent image data processing unit; determining the image data processing progress of the target image data processing unit, wherein the image data processing progress represents the processing progress of the image data processing stage currently performed by the target image data processing unit; in response to determining that the image data processing progress meets the starting condition of the image data processing stage to be performed by the subsequent image data processing unit based on the processing condition information, controlling the subsequent image data processing unit to perform the image data processing of the next image data processing stage using the image data output by the target image data processing unit.

[0006] According to another aspect of the embodiments of the present disclosure, an image data processing apparatus is provided, which comprises: a first determining module configured to determine processing condition information of a post-stage image data processing unit cascaded with a target image data processing unit, wherein the processing condition information is used to indicate a processing condition of an image data processing stage to be performed by the post-stage image data processing unit; a second determining module configured to determine an image data processing progress of the target image data processing unit, wherein the image data processing progress indicates a processing progress of an image data processing stage currently performed by the target image data processing unit; and a control module configured to, in response to determining that the image data processing progress satisfies a starting condition of the image data processing stage to be performed by the post-stage image data processing unit based on the processing condition information, control the post-stage image data processing unit to perform image data processing of a next image data processing stage using image data output by the target image data processing unit.

[0007] According to another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program configured to be executed by a processor to implement the image data processing method described above.

[0008] According to another aspect of the embodiments of the present disclosure, an electronic device is provided, which comprises: a processor; a memory configured to store processor-executable instructions; and the processor configured to read the executable instructions from the memory and execute the instructions to implement the image data processing method described above.

[0009] Based on the image data processing method, apparatus, computer readable storage medium and electronic device provided by the embodiments of the present disclosure, by determining the processing condition information of the post-stage image data processing unit cascaded with the target image data processing unit and the image data processing progress of the target image data processing unit, when it is determined that the image data processing progress satisfies the starting condition of the image data processing stage to be performed by the post-stage image data processing unit based on the processing condition information, the post-stage image data processing unit is controlled to perform image data processing of a next image data processing stage using image data output by the target image data processing unit. The post-stage image data processing unit does not need to wait for the image data processing of the pre-stage image data processing unit to be completed, but can perform image data processing as long as the image data output by the pre-stage image data processing unit satisfies the processing condition of the post-stage image data processing unit. The embodiments of the present disclosure implement the image data processing of each stage of image data processing unit in a pipeline manner, greatly reduces the time for each stage of image data processing unit to wait for the image data output by the pre-stage image data processing unit, thereby reducing the delay of image data processing and improving the speed of image data processing.

[0010] The technical solutions of the present disclosure will be further described in detail below with reference to the drawings and embodiments. Attached Figure Description

[0011] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0012] Figure 1 This is the system diagram to which this disclosure applies.

[0013] Figure 2 This is a schematic flowchart of an image data processing method provided in an exemplary embodiment of this disclosure.

[0014] Figure 3 This is an exemplary schematic diagram of at least two cascaded image data processing units according to an embodiment of the present disclosure.

[0015] Figure 4 This is a schematic flowchart of an image data processing method provided in another exemplary embodiment of this disclosure.

[0016] Figure 5 This is a schematic flowchart of an image data processing method provided in another exemplary embodiment of this disclosure.

[0017] Figure 6 This is a schematic flowchart of an image data processing method provided in another exemplary embodiment of this disclosure.

[0018] Figure 7 This is a schematic flowchart of an image data processing method provided in another exemplary embodiment of this disclosure.

[0019] Figure 8 This is a schematic diagram of the structure of an image data processing apparatus provided in an exemplary embodiment of the present disclosure.

[0020] Figure 9 This is a schematic diagram of the structure of an image data processing apparatus provided in another exemplary embodiment of this disclosure.

[0021] Figure 10 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0022] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0023] It should be noted that the relative arrangement of the components and steps, numerical expressions, and numerical values set forth in these embodiments are not meant to limit the scope of the present disclosure unless otherwise specifically indicated.

[0024] Those skilled in the art can understand that the terms "first", "second" and the like in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meaning, nor represent a logical sequence between them.

[0025] It should also be understood that in the embodiments of the present disclosure, "multiple" can mean two or more, and "at least one" can mean one, two or more.

[0026] It should also be understood that for any component, data or structure mentioned in the embodiments of the present disclosure, unless specifically limited or the context gives the opposite indication, it can be understood as one or more in general.

[0027] In addition, the term "and / or" in the present disclosure is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. In addition, the character " / " in the present disclosure generally represents an "or" relationship between the front and rear associated objects.

[0028] It should also be understood that the description of the embodiments of the present disclosure emphasizes the differences between the embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, will not be repeated.

[0029] At the same time, it should be understood that in order to facilitate the description, the size of each part shown in the drawings is not drawn according to the actual proportional relationship.

[0030] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application or uses.

[0031] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the specification where appropriate.

[0032] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0033] The embodiments of the present disclosure can be applied to terminal devices, computer systems, servers and other electronic devices, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments comprising any of the above systems, and the like.

[0034] Terminal devices, computer systems, servers and other electronic devices can be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located on local or remote computer system storage media including storage devices.

[0035] Summary

[0036] The current image data processing method usually needs to output the image data to the next link for processing after the image data processing of the previous link is completed. In the case of multiple image data processing links, the delay of image data processing is large. In order to reduce the delay of image data processing, the embodiments of the present disclosure propose an image data processing method similar to pipeline, as long as the image data output by the previous link meets the image data processing condition of the next link, the image data processing of the next link can be started, thereby reducing the delay of image data processing.

[0037] Exemplary system

[0038] Figure 1 An exemplary system architecture 100 of the image data processing method or image data processing device to which the embodiments of the present disclosure can be applied is shown.

[0039] As shown in Figure 1 The system architecture 100 can include a terminal device 101, a network 102 and a server 103. The network 102 is used to provide a communication link medium between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0040] A user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as image processing applications, video playing applications, web browser applications, instant messaging tools, etc.

[0041] The terminal device 101 can be various electronic devices, including but not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0042] The server 103 can be a server providing various services, such as a background image processing server for processing images uploaded by the terminal device 101. The background image processing server can process received image data through multiple image data processing units.

[0043] It should be noted that the image data processing method provided by the embodiments of the present disclosure can be executed by the server 103 or the terminal device 101, and correspondingly, the image data processing apparatus can be arranged in the server 103 or the terminal device 101.

[0044] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system architecture shown in FIG. 1 is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks and servers. In the case where image data does not need to be acquired remotely, the above-mentioned system architecture can not include a network, but only include a server or a terminal device.

[0045] Exemplary method

[0046] Figure 2 FIG. 1 is a flowchart of an image data processing method provided by an example embodiment of the present disclosure. The embodiment can be applied to an electronic device (such as a terminal device 101 or a server 103 as shown in FIG. 1) to perform image data processing. Figure 1 As shown in FIG. 1, the method includes the following steps: Figure 2

[0047] Step 201, determining processing condition information of a subsequent image data processing unit cascaded with a target image data processing unit.

[0048] In the embodiment, the electronic device can determine the processing condition information of the subsequent image data processing unit cascaded with the target image data processing unit. The processing condition information is used to indicate the processing condition of the image data processing stage to be performed by the subsequent image data processing unit.

[0049] ​The above-mentioned later-stage image data processing unit performs image data processing in stages. For a certain image data processing stage, when the image data processing progress of the target image data processing unit meets the processing condition of the data processing stage, the later-stage image data processing unit can perform the image data processing stage. For example, the image processing stage to be performed by the later-stage image data processing unit requires image data corresponding to 100 rows of pixels, and when the image data output by the target image data processing unit reaches the data amount corresponding to 100 rows of pixels, the later-stage image data processing unit can perform the image data processing stage.

[0050] The above-mentioned target image data processing unit and later-stage image data processing unit can be included in at least two preset image data processing units, and the at least two image data processing units are cascaded. The image data required by the later-stage image data processing unit comes from the image data output by the target image data processing unit.

[0051] As shown in FIG. 3, Figure 3 The connection relationship of five cascaded image data processing units 301 (including image data processing unit 1-image data processing unit 5) is shown. If the target image data processing unit is image data processing unit 1, the later-stage image data processing unit thereof is image data processing unit 2; if the target image data processing unit is image data processing unit 2, the later-stage image data processing unit thereof can be image data processing unit 3 or image data processing unit 5.

[0052] In an application scenario, the above-mentioned at least two image data processing units can include cascaded ISP (Image Signal Processing) processing units, FPN (Feature Pyramid Network) processing units, and NN (Neural Network) recognition processing units. The ISP processing unit is used to process the input raw image data by row, and output first image data representing color values of image pixels to the FPN processing unit; the FPN processing unit is used to process the input first image data, and output second image data representing image features to the NN recognition processing unit; and the NN recognition processing unit is used to process the input second image data, and output image recognition results (such as information of the category of a target object in the image).

[0053] It should be noted that the at least two image data processing units can be arranged in the electronic device performing the image data processing method, or can be arranged in other electronic devices connected to the electronic device. The image data processing units included in the at least two image data processing units can be hardware (for example, a neural network accelerator for performing neural network operation) or software (for example, a software module for performing ISP processing).

[0054] Optionally, as shown in Figure 3 The electronic device can be provided with a control unit 302 connected to the at least two image data processing units, for controlling the operation of the at least two image data processing units. The image data processing method can be performed by the control unit 302. The control unit 302 can be software or hardware.

[0055] In step 202, the image data processing progress of the target image data processing unit is determined.

[0056] In this embodiment, the electronic device can determine the image data processing progress of the target image data processing unit. The image data processing progress indicates the processing progress of the current image data processing stage performed by the target image data processing unit.

[0057] The image data processing progress can be represented by various types of numerical values. For example, if the target image data processing unit processes the image data corresponding to the pixels included in the image by row, the number of rows currently processed can be recorded in real time when the image data is processed, and the electronic device can use the number of rows to represent the image data processing progress of the target image data processing unit. For another example, the target image data processing unit can record the amount of data output in real time when processing the image data, and the electronic device can use the amount of data to represent the image data processing progress of the target image data processing unit.

[0058] Optionally, when the image data processing method is performed by the control unit 302, at least two counters can be arranged in the control unit 302, each counter corresponding to an image data processing unit. Each image data processing unit reports the processing progress to the control unit 302 in real time when processing the image data, and the control unit 302 records the image data processing progress of each image data processing unit by using the at least two counters.

[0059] In step 203, in response to the processing condition information, it is determined that the image data processing progress meets the start condition of the next image data processing stage to be performed by the subsequent image data processing unit, and the subsequent image data processing unit is controlled to perform the image data processing of the next image data processing stage by using the image data output by the target image data processing unit.

[0060] In this embodiment, the electronic device can determine, based on the processing condition information, whether the image data processing progress satisfies a start condition of an image data processing stage to be performed by the subsequent image data processing unit.

[0061] Specifically, the start condition of the image data processing stage represents a condition that triggers the subsequent image data processing unit to start the image data processing stage. As an example, if the processing condition information indicates that the image data processing stage to be performed by the subsequent image data processing unit requires a data amount corresponding to 100 rows of pixels, the start condition is determined to be satisfied when the data amount output by the target image processing unit reaches the data amount corresponding to 100 rows of pixels. As another example, if the data amount output by the target image processing unit reaches the data amount corresponding to 100 rows of pixels, but the subsequent image data processing unit is processing other data at this time, i.e., is in a non-idle state, the start condition of the image data processing stage to be performed by the subsequent image data processing unit is determined to be satisfied when the subsequent image data processing unit needs to be in an idle state.

[0062] In this embodiment, when it is determined that the subsequent image data processing unit satisfies the start condition of the image data processing stage to be performed, the electronic device can control the subsequent image data processing unit to perform image data processing of the next image data processing stage using the image data output by the target image data processing unit.

[0063] As an example, the electronic device can issue an instruction to the above-mentioned subsequent image data processing unit to start the next image data processing stage. Correspondingly, if the electronic device determines that the subsequent image data processing unit does not satisfy the start condition according to the image data processing progress of the target image data processing unit, an instruction is issued to the above-mentioned subsequent image data processing unit to indicate that the subsequent image data processing unit waits.

[0064] The method provided by the above embodiments of the present disclosure determines the processing condition information of the image data processing unit at the next stage cascaded with the target image data processing unit and the image data processing progress of the target image data processing unit. When the image data processing progress determined by using the processing condition information meets the starting condition of the image data processing stage to be performed by the image data processing unit at the next stage, the image data processing unit at the next stage controls the image data processing unit at the next stage to perform the image data processing of the next image data processing stage by using the image data output by the target image data processing unit. The image data processing unit at the next stage does not need to wait for the image data processing unit at the previous stage to complete the processing of the complete image data. As long as the image data output by the image data processing unit at the previous stage meets the processing condition of the image data processing unit at the next stage, the image data processing unit at the next stage can perform the image data processing. The embodiments of the present disclosure realize the image data processing of the image data processing units at all stages in a pipeline manner, greatly reduces the time for the image data processing units at all stages to wait for the image data output by the image data processing unit at the previous stage, and thus reduces the delay of the image data processing and improves the speed of the image data processing.

[0065] In some optional implementations, as shown in FIG. 2, before the step 203, the method further includes the following steps. Figure 4 As shown in FIG. 2, before the step 203, the method further includes the following steps.

[0066] In step 2031, in response to determining that the image data processing unit at the next stage meets the starting condition, the image data output by the target image data processing unit is input into the image data processing unit at the next stage.

[0067] The starting condition refers to a condition under which the state of the image data processing unit at the next stage can support the processing of the image data.

[0068] Optionally, the starting condition can include that the image data processing unit at the next stage is in the starting state and in the idle state. Correspondingly, if the image data processing unit at the next stage is in the closing state or in the busy state of processing other data, the starting condition is not met. Generally, the image data processing unit at the next stage can report the current state, so that the electronic device determines whether the image data processing unit at the next stage meets the starting condition.

[0069] The embodiment determines whether the image data processing unit at the next stage meets the starting condition. When the starting condition is met, the image data processing unit at the next stage performs the image data processing of the next image data processing stage by using the image data output by the target image data processing unit, so that the processing process of each image data processing unit is more orderly, and the risk of data link congestion caused by the flow of image data in each image data processing unit is reduced.

[0070] In some alternative implementations, such as Figure 4 As shown, in step 203, before controlling the subsequent image data processing unit to use the image data output by the target image data processing unit to perform image data processing in the next image data processing stage, step 203 further includes:

[0071] Step 2032: In response to determining that the subsequent image data processing unit does not meet the start condition, the image data output by the target image data processing unit is stored in a preset memory.

[0072] When the subsequent image data processing unit does not meet the start conditions, it means that the subsequent image data processing unit is currently unable to process the image data output by the target image data processing unit. At this time, an instruction can be sent to the target image data processing unit to instruct the target image data processing unit to store the output image data into the aforementioned preset memory.

[0073] like Figure 3 As shown, the memory 303 can be used to store image data output by multiple image data processing units 301. It should be noted that the aforementioned preset memory can be, for example... Figure 3 The memory shown can be shared by all image data processing units, or it can be a dedicated memory set up separately for the target image data processing unit.

[0074] Step 2033: In response to determining that the subsequent image data processing unit meets the start condition again, the image data output by the target image data processing unit stored in the memory is input into the subsequent image data processing unit.

[0075] After the image data output by the target image data processing unit is stored in the preset memory, if the subsequent image data processing unit meets the start condition again, it means that the subsequent image data processing unit can process the image data output by the target image data processing unit. At this time, an instruction can be issued to the subsequent image data processing unit to instruct it to retrieve the image data from the preset memory for image data processing.

[0076] This embodiment temporarily stores the image data output by the target image data processing unit in the memory when the subsequent image data processing unit does not meet the opening conditions. When the subsequent image data processing unit meets the opening conditions again, the stored image data is processed. This achieves the goal that the target image data processing unit does not need to process the image data again when the subsequent image data processing unit meets the opening conditions again, thereby improving the efficiency of image data processing.

[0077] In some optional implementations, the processing condition information is used to indicate a processing condition of the image data processing stage to be performed by the subsequent image data processing unit. The processing condition includes any one of the following conditions:

[0078] Condition one, the processing condition information indicates that the data amount of the output image data of the target image data processing unit reaches the start data amount of the subsequent image data processing unit.

[0079] The start data amount can be a fixed data amount set in advance, or a data amount determined according to the size of the actual processed image. As an example, if the subsequent image data processing unit is a processing unit for executing a feature pyramid network, the start data amount is image data corresponding to 20 rows of pixels. If the subsequent image data processing unit is a processing unit for running a neural network, the start data amount of each image data processing stage can be determined according to the size of the image. For example, when processing image data of 700 rows of pixels, the image data can be divided into four parts of 100 rows, 160 rows, 240 rows, and 200 rows, and the data amount of each part is processed each time. When the target image data processing unit outputs all image data corresponding to one part, the image data processing is started.

[0080] Condition two, the processing condition information indicates that the data amount of the output image data of the target image data processing unit reaches the start data amount of the subsequent image data processing unit, and the image region corresponding to the current image data processing stage of the target image data processing unit matches the image region corresponding to the image data processing stage to be performed by the subsequent image data processing unit.

[0081] Specifically, if the subsequent image data processing unit is to process image data corresponding to a region A in the processed image, the target image data processing unit needs to output image data not only satisfying the data amount of the image data corresponding to the region A, but also corresponding to the region A.

[0082] The processing condition of the image data processing stage provided by the embodiment can flexibly configure the start condition of the subsequent image data processing unit according to the actual processing image process, thereby helping to further improve the efficiency of image data processing.

[0083] In some optional implementations, as shown in Figure 5 Before step 201, the method further includes:

[0084] In step 501, the target image is regionally divided to obtain a set of image regions.

[0085] The target image is an image to be processed. The method of dividing the target image into regions can be set arbitrarily. For example, the target image can be divided by rows of pixels, i.e., each image region contains multiple rows of pixels. For another example, the target image can be divided by blocks, i.e., the target image is divided into multiple rectangular blocks, and each rectangular block is an image region. It should be noted that the size of each image region can be the same or different.

[0086] It should also be noted that when the method is applied to a scenario as shown in Figure 3 It should also be noted that when the method is applied to a scenario as shown in

[0087] At step 502, at least one image data processing stage of the later-stage image data processing unit is determined, and at least one image region corresponding to each image data processing stage is determined.

[0088] Specifically, each image data processing stage of the later-stage image processing unit can process image data corresponding to one or more image regions (i.e., image data corresponding to the image region output by the target image data processing unit of the previous stage).

[0089] Optionally, the size of each image region or the number of image regions corresponding to each image data processing stage (i.e., the data processing amount of each image data processing stage) can be adjusted in real time when the target image is divided according to the data processing efficiency of the later-stage image data processing unit. For example, if the data processing efficiency of the later-stage image data processing unit is high (e.g., the current processor occupancy is low, or the memory occupancy is low), the data processing amount of each image data processing stage can be increased, and vice versa, if the data processing efficiency of the later-stage image data processing unit is low, the data processing amount of each image data processing stage can be reduced.

[0090] After step 502, step 201 can include:

[0091] At step 2011, based on at least one image region corresponding to each image data processing stage, processing condition information corresponding to each image data processing stage of the later-stage image data processing unit is determined.

[0092] Specifically, for a certain image data processing stage, the corresponding processing condition information can indicate that image data corresponding to at least one image region of the image data processing stage output by the target image data processing unit has been obtained.

[0093] The embodiment divides the target image into regions in advance to determine at least one image data processing stage of the subsequent image data processing unit and an image region corresponding to each image data processing stage, so that the data amount of image data corresponding to each data processing stage of the subsequent image data processing unit is adjusted in real time according to the function and performance of the subsequent image data processing unit, thereby helping to further improve the efficiency of image data processing.

[0094] In some optional implementations, as shown in Figure 6 Step 501 includes:

[0095] Step 5011, determining an image processing mode and a minimum data requirement amount corresponding to each of the preset at least two image data processing units.

[0096] The at least two image data processing units are cascaded, and the at least two image data processing units include the target image data processing unit and the subsequent image data processing unit.

[0097] The image processing mode can include row processing or block processing. The row processing means that the image data processing unit processes the image data corresponding to each row of the target image, and the block processing means that the image data processing unit processes the image data corresponding to a certain rectangular region of the target image. The minimum data requirement amount refers to the minimum data amount required for the image data processing unit to perform image data processing once.

[0098] For example, the image processing mode of a certain image data processing unit is row processing, and the minimum data requirement amount is the image data corresponding to 8 rows of pixels.

[0099] Step 5012, dividing the target image into regions based on the image processing mode and the minimum data requirement amount to obtain a set of image regions.

[0100] As an example, when dividing the regions for a certain image data processing unit, the image data corresponding to each image region obtained by the division can be equal to the minimum data requirement amount, that is, each image data processing stage of the image data processing unit performs data processing once. Alternatively, the image data corresponding to each image region obtained by the division can be equal to an integer multiple (for example, 2 times) of the minimum data requirement amount, that is, each image data processing stage of the image data processing unit performs data processing multiple times.

[0101] This embodiment divides the target image into regions based on the image processing methods and minimum data requirements of each image data processing unit. It achieves targeted setting of the data processing volume of each image data processing unit at each image data processing stage according to the characteristics of image data processing of each unit, so that each image data processing unit can more efficiently jointly execute the processing of the target image and improve the efficiency of image data processing.

[0102] In some alternative implementations, the image data processing unit includes a first type of image data processing unit that processes images row-by-row and a second type of image data processing unit that processes images block-by-block. As an example, the image data processing unit for performing ISP and the image data processing unit for performing feature pyramid network are first type image data processing units, and the image data processing unit for performing neural network is a second type image data processing unit.

[0103] like Figure 7 As shown, step 5012 includes:

[0104] Step 50121: For the first type of image data processing unit, based on the minimum row number requirement of the first type of image data processing unit, the target image is divided into row regions to obtain a set of first image regions.

[0105] Optionally, the least common multiple of the minimum row requirement of each of the first type of image data processing units can be taken, and the target image can be divided according to the least common multiple. For example, if the minimum row requirement of image data processing unit a is 8 rows and the minimum row requirement of image data processing unit b is 12 rows, then the least common multiple of 8 and 12 is 24, and each divided image region includes 24 rows of pixels. For image data processing unit a, each image data processing stage requires 3 calculations; for image data processing unit b, each image data processing stage requires 2 calculations.

[0106] Optionally, the target image can be divided directly according to the minimum row requirement of each image data processing unit of the first type. For example, for an image data processing unit with a minimum row requirement of 8 rows, each divided image region includes 8 rows of pixels; for an image data processing unit with a minimum row requirement of 12 rows, each divided image region includes 12 rows of pixels.

[0107] Step 50122: For the second type of image data processing unit, based on the minimum data block requirement of the second type of data processing unit, the target image is divided into block regions to obtain a set of second image regions.

[0108] Optionally, the target image can be divided according to the maximum minimum data block requirement of each of the second-type image data processing units, such that the size of each resulting image region is greater than or equal to the block size corresponding to that maximum value. For example, if the minimum data block requirement of image data processing unit c is 16×16 and the minimum data block requirement of image data processing unit d is 32×32, then the size of each divided image region is greater than or equal to 32×32. For image data processing unit c, each image data processing stage requires four calculations.

[0109] This embodiment divides the image region based on the minimum row requirement or minimum data block requirement of multiple image data processing units of the same type. This enables the target image to be uniformly divided for multiple image data processing units of the same type. The same type of data processing units process the image data corresponding to the same size image region, which helps to reduce the number of data processing stages of each image data processing unit and the number of data exchanges, thereby helping to improve the efficiency of image data processing.

[0110] Exemplary apparatus

[0111] Figure 8 This is a schematic diagram of the structure of an image data processing apparatus provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 8 As shown, the image data processing device includes: a first determining module 801, used to determine processing condition information of a subsequent image data processing unit cascaded with the target image data processing unit, wherein the processing condition information is used to indicate the processing conditions of the image data processing stage to be performed by the subsequent image data processing unit; a second determining module 802, used to determine the image data processing progress of the target image data processing unit, wherein the image data processing progress indicates the processing progress of the current image data processing stage performed by the target image data processing unit; and a control module 803, used to respond to determining, based on the processing condition information, that the image data processing progress meets the start conditions of the image data processing stage to be performed by the subsequent image data processing unit, and to control the subsequent image data processing unit to perform image data processing of the next image data processing stage using the image data output by the target image data processing unit.

[0112] In this embodiment, the first determining module 801 can determine the processing condition information of the subsequent image data processing unit cascaded with the target image data processing unit. The processing condition information indicates the processing conditions of the image data processing stage to be performed by the subsequent image data processing unit.

[0113] The above-mentioned later-stage image data processing unit performs image data processing in stages. For a certain image data processing stage, when the image data processing progress of the target image data processing unit meets the processing condition of the data processing stage, the later-stage image data processing unit can perform the image data processing stage. For example, the image processing stage to be performed by the later-stage image data processing unit requires image data corresponding to 100 rows of pixels, and when the image data output by the target image data processing unit reaches the data amount corresponding to 100 rows of pixels, the later-stage image data processing unit can perform the image data processing stage.

[0114] The above-mentioned target image data processing unit and later-stage image data processing unit can be included in at least two preset image data processing units, and the at least two image data processing units are cascaded. The image data required by the later-stage image data processing unit comes from the image data output by the target image data processing unit.

[0115] It should be noted that the above-mentioned at least two image data processing units can be arranged in the above-mentioned electronic device, or can be arranged in other electronic devices connected to the above-mentioned electronic device. The image data processing units included in the above-mentioned at least two image data processing units can be hardware (for example, a neural network accelerator for performing neural network operation) or software (for example, a software module for performing ISP processing).

[0116] In this embodiment, the second determination module 802 can determine the image data processing progress of the target image data processing unit. The image data processing progress indicates the processing progress of the image data processing stage currently performed by the target image data processing unit.

[0117] The image data processing progress can be represented by various types of numerical values. For example, if the target image data processing unit processes image data corresponding to pixels included in an image by row, the number of rows currently processed can be recorded in real time when processing image data, and the second determination module 802 can use the number of rows to represent the image data processing progress of the target image data processing unit. For another example, the target image data processing unit can record the data amount of the data output in real time when processing image data, and the second determination module 802 can use the data amount to represent the image data processing progress of the target image data processing unit.

[0118] In this embodiment, the control module 803 can determine whether the image data processing progress meets the start condition of the image data processing stage to be performed by the later-stage image data processing unit based on the processing condition information.

[0119] Specifically, the start condition of the image data processing stage represents a condition that triggers the start of the image data processing stage by the subsequent image data processing unit. As an example, if the processing condition information indicates that the image data processing stage to be performed by the subsequent image data processing unit requires a data amount corresponding to 100 rows of pixels, the start condition is determined to be satisfied when the data amount output by the target image processing unit reaches the data amount corresponding to 100 rows of pixels. As another example, if the data amount output by the target image processing unit reaches the data amount corresponding to 100 rows of pixels, but the subsequent image data processing unit is currently processing other data, i.e., is in a non-idle state, the start condition of the image data processing stage to be performed by the subsequent image data processing unit is determined to be satisfied when the subsequent image data processing unit is in an idle state.

[0120] In the present embodiment, when it is determined that the start condition of the image data processing stage to be performed by the subsequent image data processing unit is satisfied, the control module 803 can control the subsequent image data processing unit to perform image data processing of the next image data processing stage using the image data output by the target image processing unit.

[0121] As an example, the control module 803 can issue an instruction to the above-mentioned subsequent image data processing unit for starting the next image data processing stage. Correspondingly, if the control module 803 determines that the start condition is not satisfied by the subsequent image data processing unit according to the image data processing progress of the target image processing unit, an instruction is issued to the above-mentioned subsequent image data processing unit for instructing the subsequent image data processing unit to wait.

[0122] Referring to Figure 9 , Figure 9 is a structural schematic diagram of an image data processing apparatus provided by another exemplary embodiment of the present disclosure.

[0123] In some optional implementations, the control module 803 comprises a first input unit 8031 configured to input the image data output by the target image processing unit to the subsequent image data processing unit in response to a determination that the start condition is satisfied by the subsequent image data processing unit.

[0124] In some optional implementations, the start condition comprises that the subsequent image data processing unit is in an enabled state and in an idle state.

[0125] In some optional implementation, the control module 803 further includes: a storage unit 8032, configured to store the image data output by the target image data processing unit into a preset storage in response to the determination that the post-stage image data processing unit does not satisfy the starting condition; and a second input unit 8033, configured to input the image data output by the target image data processing unit stored in the storage into the post-stage image data processing unit in response to the determination that the post-stage image data processing unit again satisfies the starting condition.

[0126] In some optional implementation, the processing condition information is used to indicate the processing condition of the image data processing stage to be performed by the post-stage image data processing unit, and the processing condition includes: the processing condition information indicates that the data amount of the output image data of the target image data processing unit reaches the starting data amount of the post-stage image data processing unit; or the processing condition information of the target image data processing unit indicates that the data amount of the output image data of the target image data processing unit reaches the starting data amount of the post-stage image data processing unit, and the image region corresponding to the current image data processing stage of the target image data processing unit matches the image region corresponding to the image data processing stage to be performed by the post-stage image data processing unit.

[0127] In some optional implementation, the apparatus further includes: a division module 804, configured to divide the target image into a set of image regions; a third determination module 805, configured to determine at least one image data processing stage of the post-stage image data processing unit, and at least one image region corresponding to the at least one image data processing stage respectively; and the first determination module 801 is further configured to: determine the processing condition information of the post-stage image data processing unit corresponding to the at least one image data processing stage respectively based on the at least one image region corresponding to the at least one image data processing stage respectively.

[0128] In some optional implementation, the division module 804 includes: a determination unit 8041, configured to determine an image processing mode and a minimum data requirement amount corresponding to each of at least two preset image data processing units respectively, wherein the at least two image data processing units are cascaded, and the at least two image data processing units include the target image data processing unit and the post-stage image data processing unit; and a division unit 8042, configured to divide the target image into the set of image regions based on the image processing mode and the minimum data requirement amount.

[0129] In some optional implementations, the image data processing unit includes a first type of image data processing unit with a row-based image processing mode, and a second type of image data processing unit with a block-based image processing mode; the division unit 8042 includes a first division sub-unit 80421 configured to perform row-based region division on the target image based on a minimum row quantity requirement of the first type of image data processing unit, to obtain a first set of image regions; and a second division sub-unit 80422 configured to perform block-based region division on the target image based on a minimum data block requirement of the second type of image data processing unit, to obtain a second set of image regions.

[0130] The image data processing apparatus provided by the above embodiments of the present disclosure determines the processing condition information of the image data processing unit at a later stage cascaded with the target image data processing unit, and the image data processing progress of the target image data processing unit. When the image data processing progress determined by using the processing condition information satisfies the starting condition of the image data processing stage to be performed by the image data processing unit at the later stage, the image data processing unit at the later stage is controlled to perform image data processing of the next image data processing stage by using the image data output by the target image data processing unit. The image data processing unit at the later stage does not need to wait for the image data processing unit at the previous stage to complete processing of the complete image data. As long as the image data output by the image data processing unit at the previous stage meets the processing condition of the image data processing unit at the later stage, the image data processing unit at the later stage can perform image data processing. The embodiments of the present disclosure implement the image data processing units at all stages to process image data in a pipeline manner, greatly reducing the time for the image data processing units at all stages to wait for the image data output by the image data processing unit at the previous stage, thereby reducing the delay of image data processing and improving the speed of image data processing.

[0131] Exemplary electronic device

[0132] Hereinafter, an electronic device according to an embodiment of the present disclosure will be described with reference to the accompanying drawings. Figure 10 The electronic device can be any one or both of a terminal device 101 and a server 103 as illustrated in Figure 1 , or a single device independent of them, which can communicate with the terminal device 101 and the server 103 to receive input signals collected therefrom.

[0133] Figure 10 FIG. 1 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure.

[0134] As illustrated in Figure 10 , the electronic device 1000 includes one or more processors 1001 and a memory 1002.

[0135] The processor 1001 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction executing capabilities, and can control other components in the electronic device 1000 to perform desired functions.

[0136] The memory 1002 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile and / or non-volatile memory. For example, the volatile memory can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory can include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 1001 can execute the program instructions to implement the image data processing method of various embodiments of the present disclosure described above and / or other desired functions. Various contents such as image data can also be stored in the computer-readable storage media.

[0137] The electronic device 1000 further includes at least two image data processing units 1003, which can be hardware processing units or software processing units. The at least two image data processing units 1003 can be disposed in the processor 1001 or outside the processor 1001. The memory 1002 can also store image data output by the at least two image data processing units 1003, and the at least two image data processing units 1003 can extract image data to be processed from the memory 1002.

[0138] In one example, the electronic device 1000 can further include an input device 1004 and an output device 1005, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0139] For example, when the electronic device is the terminal device 101 or the server 103, the input device 1004 can be a camera, a mouse, a keyboard, and the like, for inputting image data, various commands, and the like. When the electronic device is a stand-alone device, the input device 1004 can be a communication network connector for receiving image data, various commands, and the like input from the terminal device 101 and the server 103.

[0140] The output device 1005 can output various information including processed image data to the outside. The output device 1005 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0141] Of course, in order to simplify, Figure 10Only some of the components of the electronic device 1000 related to the present disclosure are shown, and components such as a bus, an input / output interface, and the like are omitted. In addition thereto, the electronic device 1000 can further include any other appropriate components according to a specific application.

[0142] Exemplary computer program product and computer readable storage medium

[0143] In addition to the above-mentioned methods and devices, embodiments of the present disclosure can also be a computer program product including computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the image data processing method according to various embodiments of the present disclosure described in the above "Exemplary Methods" section of the present specification.

[0144] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0145] In addition, embodiments of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the image data processing method according to various embodiments of the present disclosure described in the above "Exemplary Methods" section of the present specification.

[0146] The computer readable storage medium can be any combination of one or more non-transitory media. The non-transitory medium can be a non-transitory signal medium or a non-transitory storage medium. The non-transitory storage medium can include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the non-transitory storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0147] The above describes the basic principles of the present disclosure in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present disclosure are merely examples and are not limiting, and these advantages, benefits, effects and the like cannot be considered as necessary for each embodiment of the present disclosure. In addition, the above specific details are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present disclosure to be necessarily implemented with the above specific details.

[0148] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between each embodiment can be mutually referred to. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0149] The block diagrams of the devices, apparatuses, equipment, systems involved in the present disclosure are only exemplary examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, which mean "including but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0150] The methods and devices of the present disclosure can be implemented in many ways. For example, the methods and devices of the present disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present disclosure are not limited to the above specific description, unless otherwise specifically described. In addition, in some embodiments, the present disclosure can also be implemented as programs recorded in recording media, which include machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers the recording media storing the programs for executing the method according to the present disclosure.

[0151] It should also be noted that in the devices, equipment and methods of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present disclosure.

[0152] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0153] The above description has been presented for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the disclosure to the forms disclosed herein. Although several example aspects and embodiments have been discussed above, those of ordinary skill in the art will appreciate a variety of modifications, alternatives, permutations, additions, and sub-combinations of the described aspects and features.

Claims

1. A method for processing image data, comprising: determining processing condition information of a post-stage image data processing unit cascaded with a target image data processing unit, wherein the processing condition information is used to indicate a processing condition of an image data processing stage to be performed by the post-stage image data processing unit; determining an image data processing progress of the target image data processing unit, wherein the image data processing progress indicates a processing progress of an image data processing stage currently performed by the target image data processing unit; in response to determining, based on the processing condition information, that the image data processing progress satisfies a start condition of the image data processing stage to be performed by the post-stage image data processing unit, controlling the post-stage image data processing unit to perform image data processing of a next image data processing stage using image data output by the target image data processing unit; wherein the processing condition information used to indicate the processing condition of the image data processing stage to be performed by the post-stage image data processing unit comprises: the processing condition information indicates that a data amount of the image data output by the target image data processing unit reaches a start data amount of the post-stage image data processing unit; or the processing condition information indicates that the data amount of the image data output by the target image data processing unit reaches the start data amount of the post-stage image data processing unit, and an image region corresponding to a current image data processing stage of the target image data processing unit matches an image region corresponding to the image data processing stage to be performed by the post-stage image data processing unit.

2. The method of claim 1, wherein, Before the controlling the post-stage image data processing unit to perform image data processing of the next image data processing stage using the image data output by the target image data processing unit, the method further comprises: in response to determining that the post-stage image data processing unit satisfies an open condition, inputting the image data output by the target image data processing unit into the post-stage image data processing unit.

3. The method of claim 2, wherein, the open condition comprises: the post-stage image data processing unit is in an open state and in an idle state.

4. The method of claim 2 or 3, wherein, Before the controlling the post-stage image data processing unit to perform image data processing of the next image data processing stage using the image data output by the target image data processing unit, the method further comprises: in response to determining that the post-stage image data processing unit does not satisfy the open condition, storing the image data output by the target image data processing unit in a preset storage; and in response to determining that the post-stage image data processing unit again satisfies the open condition, inputting the image data output by the target image data processing unit stored in the storage into the post-stage image data processing unit.

5. The method of claim 1, wherein, Before the determining the processing condition information of the post-stage image data processing unit cascaded with the target image data processing unit, the method further comprises: dividing a target image into a set of image regions; determining at least one image data processing stage of the post-stage image data processing unit and at least one image region corresponding to the at least one image data processing stage, respectively; The processing condition information of the post-stage image data processing unit cascaded with the target image data processing unit is determined, including: Based on the at least one image area corresponding to the at least one image data processing stage respectively, the processing condition information of the post-stage image data processing unit in the at least one image data processing stage respectively is determined.

6. The method of claim 5, wherein, The target image is regionally divided to obtain a set of image regions, including: Determine the image processing mode and the minimum data requirement amount corresponding to the at least two image data processing units respectively, wherein the at least two image data processing units are cascaded, and the at least two image data processing units include the target image data processing unit and the post-stage image data processing unit; Based on the image processing mode and the minimum data requirement amount, the target image is regionally divided to obtain a set of image regions.

7. The method of claim 6, wherein, The image data processing unit includes a first type of image data processing unit with a row processing image processing mode and a second type of image data processing unit with a block processing image processing mode; Based on the image processing mode and the minimum data requirement amount, the target image is regionally divided to obtain a set of image regions, including: For the first type of image data processing unit, based on the minimum row number requirement amount of the first type of image data processing unit, the target image is regionally divided by rows to obtain a set of first image regions; For the second type of image data processing unit, based on the minimum data block requirement amount of the second type of data processing unit, the target image is regionally divided by blocks to obtain a set of second image regions.

8. An image data processing apparatus, comprising: A first determination module for determining the processing condition information of the post-stage image data processing unit cascaded with the target image data processing unit, wherein the processing condition information is used to represent the processing condition of the image data processing stage to be performed by the post-stage image data processing unit; A second determination module for determining the image data processing progress of the target image data processing unit, wherein the image data processing progress represents the processing progress of the image data processing stage currently performed by the target image data processing unit; A control module for, in response to determining that the image data processing progress meets the starting condition of the image data processing stage to be performed by the post-stage image data processing unit based on the processing condition information, controlling the post-stage image data processing unit to perform the image data processing of the next image data processing stage using the image data output by the target image data processing unit. The processing condition information used to represent the processing condition of the image data processing stage to be performed by the post-stage image data processing unit includes: The processing condition information represents that the data amount of the output image data of the target image data processing unit reaches the starting data amount of the post-stage image data processing unit; or The processing condition information indicates that a data amount of output image data of the target image data processing unit reaches a start data amount of the subsequent image data processing unit, and an image region corresponding to a current image data processing stage of the target image data processing unit matches an image region corresponding to an image data processing stage to be performed by the subsequent image data processing unit. 9.A computer readable storage medium, the storage medium storing a computer program, the computer program being configured to be executed by a processor to implement the method of any one of claims 1-7. 10.An electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-7.

Images