Image processing method, device, system, electronic equipment and readable storage medium

Abstract

The application relates to an image processing method, device, system, electronic equipment, storage medium and computer program product. The method comprises the following steps: acquiring image data information of a first frame image; the first frame image is obtained by a first processor; performing depth information processing based on the image data information of the first frame image to obtain depth information of the first frame image; based on receiving a second frame image by the first processor, and before the first processor finishes receiving the second frame image, updating the depth information of the first frame image into metadata of the second frame image; wherein the metadata of the second frame image is used for being sent to a receiving end by the first processor, so as to instruct the receiving end to perform processing on the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image. The method can improve the accuracy of the obtained depth information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an image processing method, device, system, electronic device and computer readable storage medium. BACKGROUND

[0002] With the development of image processing technology, depth images have been applied in various fields, such as image segmentation, edge detection, image registration, three-dimensional reconstruction and image virtualization, etc. Depth images directly reflect the geometric shape of the visible surface of the scene, which can improve the image processing effect. For example, using the depth data in the depth image to perform virtualization processing on the image can make the virtualized image more natural and improve the image virtualization effect. However, the image data output by the photosensitive element is prone to mismatch with the image data actually used for depth estimation, resulting in low accuracy of the depth information obtained by depth estimation. SUMMARY

[0003] The embodiments of the present application provide an image processing method, device, system, electronic device and computer readable storage medium, which can improve the accuracy of the obtained depth information.

[0004] An image processing method applied to a second processor, the method comprising:

[0005] obtaining image data information of a first frame image; the first frame image is obtained by a first processor;

[0006] performing depth information processing based on the image data information of the first frame image to obtain depth information of the first frame image;

[0007] based on receiving a second frame image by the first processor, and before the first processor finishes receiving the second frame image, updating the depth information of the first frame image into metadata of the second frame image;

[0008] wherein the metadata of the second frame image is used to be sent to a receiving end by the first processor, to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0009] An image processing device, the device comprising:

[0010] an image data information obtaining module, configured to obtain image data information of a first frame image; the first frame image is obtained by a first processor;

[0011] a depth information processing module, configured to perform depth information processing based on the image data information of the first frame image to obtain depth information of the first frame image;

[0012] a depth information updating module, configured to update the depth information of the first frame image into metadata of the second frame image based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image;

[0013] The metadata of the second frame image is used to be sent to a receiving end by the first processor to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0014] An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above image processing method when executing the computer program.

[0015] A computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program implements the steps of the above image processing method when executed by a processor.

[0016] A computer program product, the computer program product includes a computer program, and the computer program implements the steps of the above image processing method when executed by a processor.

[0017] The above image processing method, device, electronic device, storage medium and computer program product, the second processor obtains image data information of the first frame image received by the first processor, performs depth information processing based on the image data information of the first frame image, updates the obtained depth information of the first frame image into metadata of the second frame image based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image, and the metadata of the second frame image instructs the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image after being sent to the receiving end by the first processor. In the image processing process, the second processor processes the image data information each time, which is the image data information of the image received by the first processor, ensuring the consistency of the image data information of the same image in different processing processes, thereby improving the accuracy of the obtained depth information.

[0018] An image processing method, the method comprising:

[0019] receiving a first frame image by a first processor;

[0020] obtaining image data information of the first frame image by a second processor to perform depth information processing based on the image data information of the first frame image by the second processor, and updating the obtained depth information of the first frame image into metadata of a second frame image based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image.

[0021] The metadata of the second frame image is sent to a receiving end by the first processor to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0022] An image processing apparatus, the apparatus comprising:

[0023] A first frame image receiving module configured to receive a first frame image by a first processor;

[0024] A depth information processing module configured to acquire image data information of the first frame image by a second processor, to process the depth information based on the image data information of the first frame image by the second processor, and to update the obtained depth information of the first frame image into metadata of a second frame image based on receiving the second frame image by the first processor and before the end of receiving the second frame image by the first processor;

[0025] A metadata sending module configured to send the metadata of the second frame image to a receiving end by the first processor to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0026] An electronic device, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the above image processing method when executing the computer program.

[0027] A computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program implementing the steps of the above image processing method when executed by a processor.

[0028] A computer program product, the computer program product comprising a computer program, and the computer program implementing the steps of the above image processing method when executed by a processor.

[0029] The image processing method, device, electronic device, storage medium and computer program product described above, by the second processor, obtain image data information of the first frame image received by the first processor, perform depth information processing based on the image data information of the first frame image, and based on receiving the second frame image by the first processor and before the end of receiving the second frame image by the first processor, update the obtained depth information of the first frame image into the metadata of the second frame image, and send the metadata of the second frame image to the receiving end by the first processor, to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image. In the image processing process, the second processor each time processes the image data information of the image, which is the image data information of the image received by the first processor, ensuring the consistency of the image data information of the same image in different processing processes, thereby improving the accuracy of the obtained depth information and improving the image processing effect when processing based on the depth information.

[0030] An image processing method applied to a first processor, the method comprising:

[0031] receiving a first frame image and sending the first frame image to a receiving end;

[0032] based on receiving a second frame image, sending metadata of the second frame image to the receiving end to instruct the receiving end to process the corresponding first frame image according to depth information of the first frame image in the metadata of the second frame image;

[0033] wherein the depth information of the first frame image is obtained by a second processor based on depth information processing of image data information of the first frame image, and is updated by the second processor into the metadata of the second frame image based on receiving the second frame image and before the end of receiving the second frame image.

[0034] An image processing device, the device comprising:

[0035] a first frame image receiving module configured to receive a first frame image and send the first frame image to a receiving end;

[0036] a second frame image metadata sending module configured to, based on receiving a second frame image, send metadata of the second frame image to the receiving end to instruct the receiving end to process the corresponding first frame image according to depth information of the first frame image in the metadata of the second frame image;

[0037] The depth information of the first frame image is obtained by the second processor based on image data information of the first frame image, and the second processor updates the obtained depth information of the first frame image into metadata of the second frame image before receiving the second frame image ends.

[0038] An electronic device includes a memory storing a computer program and a processor implementing the steps of the above image processing method when executing the computer program.

[0039] A computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the above image processing method.

[0040] A computer program product including a computer program, the computer program being executed by a processor to implement the steps of the above image processing method.

[0041] The above image processing method, device, electronic device, storage medium and computer program product, the first processor receives the first frame image and sends the first frame image to the receiving end, based on receiving the second frame image, the metadata of the second frame image is sent to the receiving end to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image. Wherein, the depth information of the first frame image is obtained by the second processor based on the image data information of the first frame image, and the second processor updates the obtained depth information of the first frame image into the metadata of the second frame image before receiving the second frame image ends. In the image processing process, the second processor is each time for the image data information processed by the depth information, which is the image data information of the image received by the first processor, to ensure the consistency of the image data information of the same image in different processing processes, thereby improving the accuracy of the obtained depth information, and improving the image processing effect when processing based on the depth information.

[0042] An image processing system, the system comprising:

[0043] The first processor is configured to receive a first frame image, and based on receiving a second frame image, send metadata of the second frame image to a receiving end to instruct the receiving end to process the corresponding first frame image according to depth information of the first frame image in the metadata of the second frame image.

[0044] The second processor is configured to acquire image data information of the first frame image, perform depth information processing based on the image data information of the first frame image, and obtain depth information of the first frame image; and based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image, update the obtained depth information of the first frame image into metadata of the second frame image.

[0045] The scheduling processor is configured to control the first processor to receive the first frame image and the second frame image, and control the first processor to send the metadata of the second frame image to the receiving end.

[0046] The scheduling processor is further configured to control the second processor to perform depth information processing based on the image data information of the first frame image, and control the second processor to update the depth information of the first frame image into the metadata of the second frame image.

[0047] In the image processing system, the scheduling processor controls the first processor to receive the first frame image and send the first frame image to the receiving end, and based on receiving the second frame image, sends the metadata of the second frame image to the receiving end to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image. The scheduling processor controls the second processor to acquire the image data information of the first frame image received by the first processor, perform depth information processing based on the image data information of the first frame image, and based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image, update the obtained depth information of the first frame image into the metadata of the second frame image. In the image processing process, the second processor processes the image data information each time, which is the image data information of the image received by the first processor, ensuring the consistency of the image data information of the same image in different processing processes, thereby improving the accuracy of the obtained depth information and improving the image processing effect when processing based on the depth information. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0049] Figure 1 It is an application environment diagram of the image processing method in one embodiment;

[0050] Figure 2 It is a flowchart of the image processing method in one embodiment;

[0051] Figure 3 a flowchart of depth information processing steps in one embodiment;

[0052] Figure 4 a flowchart of an image processing method in another embodiment;

[0053] Figure 5 a timing diagram of an image processing method in one embodiment;

[0054] Figure 6 a flowchart of an image processing method in yet another embodiment;

[0055] Figure 7 a schematic diagram of a blurring process in one embodiment;

[0056] Figure 8 a schematic diagram of a scheduling process of an image processing method in one embodiment;

[0057] Figure 9 a timing diagram of an image processing method in another embodiment;

[0058] Figure 10 a structural block diagram of an image processing apparatus in one embodiment;

[0059] Figure 11 a structural block diagram of an image processing apparatus in another embodiment;

[0060] Figure 12 a structural block diagram of an image processing apparatus in yet another embodiment;

[0061] Figure 13 a structural schematic block diagram of an image processing system in one embodiment;

[0062] Figure 14 an internal structural diagram of an electronic device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0064] The image processing method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment shown. Among them, the scheduling processor 104 respectively communicates with the first processor 102 and the second processor 106, and can communicate through the network. In addition, the first processor 102 and the second processor 106 can also communicate, such as through the network. The data storage system can store the data required to be processed by the corresponding processor. The data storage system can be integrated on the corresponding processor, or placed on the cloud or other network server. In application, image processing can be realized based on the communication between the first processor 102 and the second processor 106. Specifically, the second processor 106 obtains the image data information of the first frame image received by the first processor 102, and the second processor 106 processes the depth information based on the image data information of the first frame image. In the case of receiving the second frame image through the first processor 102, and before the first processor 102 receives the second frame image, the second processor 106 updates the obtained depth information of the first frame image to the metadata of the second frame image. The metadata of the second frame image is sent to the receiving end through the first processor 102, and the receiving end is instructed to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0065] In the implementation of the image processing method based on the first processor 102, the first processor 102 receives the first frame image and sends the first frame image to the receiving end. The first processor 102 sends the metadata of the second frame image to the receiving end based on receiving the second frame image, so as to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image. Among them, the depth information of the first frame image is obtained by the second processor 106 based on the depth information processing of the image data information of the first frame image, and the second processor 106 updates the obtained depth information of the first frame image to the metadata of the second frame image based on the first processor 102 receiving the second frame image and before the first processor 102 receiving the second frame image ends.

[0066] In the implementation of the image processing method based on the scheduling processor 104, the scheduling processor 104 obtains the image data information of the first frame image received by the first processor 102 through the second processor 106, and the second processor 106 processes the depth information based on the image data information of the first frame image. In the case of receiving the second frame image through the first processor 102, and before the first processor 102 receives the second frame image ends, the second processor 106 updates the obtained depth information of the first frame image to the metadata of the second frame image, and the scheduling processor 104 sends the metadata of the second frame image to the receiving end through the first processor 102, so as to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0067] The first processor 102, the scheduling processor 104, and the second processor 106 can be processors in an electronic device, which can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, and the like. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, and the like. The first processor 102, the scheduling processor 104, and the second processor 106 can also be implemented by a server, which can be implemented by a stand-alone server or a server cluster composed of multiple servers.

[0068] In one embodiment, as shown in Figure 2 , an image processing method is provided. The method is described by taking the second processor in Figure 1 as an example. The second processor can be a processor in an electronic device or a server. In this embodiment, the method includes the following steps:

[0069] In step 202, image data information of a first frame image is obtained. The first frame image is received by the first processor.

[0070] The first frame image is received by the first processor, and specifically can be received by the first processor from a photosensitive element of a camera. When the camera is shooting, imaging is realized by the photosensitive element, and specifically the captured light signal is converted into an electrical signal that can be processed. When the camera is shooting, the photosensitive element of the camera sends the captured image to the first processor, which receives and further processes the image, such as linear correction, noise removal, bad point removal, interpolation, white balance, automatic exposure control, and the like, to improve the imaging quality. The image captured by the camera is sent frame by frame to the first processor by the photosensitive element, and the first processor receives each frame of image. The first frame image is the image currently received by the first processor from the photosensitive element of the camera. The image data information refers to information related to the first frame image, which specifically can include attribute information, storage location information, and the like of the first frame image, and can also include specific image data of the first frame image. The attribute information can be description information for the first frame image, which can include, but is not limited to, width, height, or bit width of the first frame image, and the like.

[0071] Specifically, when the first frame image needs to be processed by depth information, such as when depth estimation needs to be performed on the first frame image, the second processor acquires image data information of the first frame image, which can include attribute information and storage location information of the first frame image. Based on the storage location information, the second processor can acquire image data of the first frame image, and perform depth estimation based on the image data and the attribute information of the first frame image to obtain depth information of the first frame image. The image data information can also include image data of the first frame image, so that the second processor can directly perform depth information processing according to the image data in the image data information. In a specific implementation, the second processor can trigger acquisition of the image data information of the first frame image when the first processor ends receiving the first frame image, i.e., when the first processor completes receiving the first frame image, to process the image data information of the first frame image. For example, the second processor can acquire the image data information of the first frame image from the first processor when it is determined that the first processor ends receiving the first frame image, or the second processor can acquire the image data information of the first frame image from the memory.

[0072] In step 204, depth information processing is performed based on the image data information of the first frame image to obtain depth information of the first frame image.

[0073] The depth information can include depth data of an image, and specifically can include a depth image. The depth image refers to an image in which distances from an image collector to each point in a scene, i.e., depth, are taken as pixel values, and directly reflects the geometric shape of a visible surface of a scene. The depth image can be calculated as point cloud data through coordinate conversion, and point cloud data with regular and necessary information can also be calculated back to depth image data. In each image frame provided by a depth data stream, each pixel point represents the distance from the object at a specific coordinate in the field of view of a depth sensor to the nearest object to the camera plane. In a specific application, the depth information can also include description information of the depth data of the image, i.e., the depth information can also include attribute information of the depth data, such as various description information of the depth image, such as width, height, and bit width. Based on the depth information of the first frame image, various processing can be performed on the first frame image, such as image segmentation, edge detection, image registration, three-dimensional reconstruction, and image blurring.

[0074] Specifically, the second processor performs depth information processing on the image data information of the first frame image, and specifically can perform depth estimation based on the image data information of the first frame image to obtain depth information of the first frame image. In a specific implementation, the second processor can perform depth calculation on the image data information of the first frame image based on a depth learning algorithm to obtain depth information of the first frame image, such as depth data of the first frame image.

[0075] At step 206, based on receiving the second frame image by the first processor, and before the end of receiving the second frame image by the first processor, the depth information of the first frame image is updated into the metadata of the second frame image; wherein the metadata of the second frame image is used for sending to the receiving end by the first processor, so as to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0076] The second frame image is the next adjacent received image after the first frame image is received, that is, when the receiving is triggered again after the end of receiving the first frame image, the second frame image is received. Metadata, also known as intermediate data and relay data, is data for describing data, mainly information for describing data attributes. Metadata can be used to support functions such as indicating storage location, historical data, resource search, file recording, etc. The metadata of the image is data for describing the image, which can be used as attribute information of the image. The metadata of the image can include EXIF (Exchangeable Image File format), IPTC (International Press Telecommunications Council) and XMP (Extensible Metadata Platform) types. Among them, EXIF is usually automatically added by a digital camera when taking a photo, such as camera model, lens, exposure, picture size, etc.; IPTC can include picture title, keyword, description, author, copyright, etc.; XMP is a standard for metadata storage and management. The metadata of the image can be obtained by the first processor when receiving the image. The receiving end is used for processing the image, such as image segmentation, image blurring or three-dimensional reconstruction, etc.

[0077] Specifically, after completing the reception of the first frame image, the first processor can continue to receive the next frame image from the photosensitive element, that is, to continue to receive the second frame image, and the photosensitive element needs to go through a vertical blanking period for the next frame after processing and outputting one frame of image data, that is, after the scanning point of the photosensitive element scans one frame, it returns from the lower right corner of the image to the upper left corner of the image to start scanning a new frame. The vertical blanking period is the time interval from the end of reading one frame by the photosensitive element to the start of reading the next frame. In the time interval corresponding to the vertical blanking period, the photosensitive element does not output image data, and the first processor also does not receive image data. After going through the vertical blanking period, the photosensitive element outputs the second frame image, and the first processor also starts to receive the second frame image.

[0078] The second processor updates the depth information of the first frame image into the metadata of the second frame image when determining that the first processor receives the second frame image. Specifically, the second processor can determine the storage location of the metadata of the second frame image, and update the depth information of the first frame image into the storage location of the metadata of the second frame image. When the second processor updates the depth information of the first frame image, it needs to complete the processing of updating the depth information of the first frame image into the metadata of the second frame image before the first processor finishes receiving the second frame image, so as to ensure that the depth information of the first frame image can also be sent together when the first processor sends the metadata of the second frame image to the receiving end, thereby ensuring the processing efficiency of the metadata of the second frame image.

[0079] For the metadata of the second frame image, the first processor can send it to the receiving end, so that the receiving end obtains the depth information of the first frame image from the metadata of the second frame image after obtaining the metadata of the second frame image, and processes the corresponding first frame image according to the depth information of the first frame image, such as image segmentation, image blurring, etc. Wherein, the first processor can also send the first frame image to the receiving end, specifically, the first processor can send the received first frame image to the receiving end after completing the reception of the first frame image.

[0080] In one specific application, when the first processor completes the reception of the first frame image, i.e. when the first processor finishes receiving the first frame image, the second processor obtains the image data information of the first frame image in response to the end of the reception of the first frame image by the first processor, and performs depth estimation based on the image data information of the first frame image to obtain the depth information of the first frame image, which can specifically include depth data of the first frame image, and can also include attribute data of the depth data. When the first processor triggers the reception of the second frame image, the second processor updates the depth information of the first frame image into the metadata of the second frame image in response to the reception of the second frame image by the first processor, and completes the update processing of the depth information of the first frame image, i.e. the processing of updating the depth information of the first frame image into the metadata of the second frame image, before the first processor finishes receiving the second frame image. The first processor can send the metadata of the second frame image to the receiving end, and the receiving end can extract the depth information of the first frame image from the metadata of the second frame image, and process the corresponding first frame image according to the depth information of the first frame image, such as three-dimensional reconstruction, blurring processing, etc.

[0081] In the specific application, the second processor processes the depth information of the first frame image, including calculating the depth information of the first frame image and updating the calculated depth information of the first frame image to the metadata of the second frame image, both of which are performed between the end of the first frame image received by the first processor and the end of the second frame image received by the first processor, i.e. the first processor has completed the reception of the first frame image and will not change the image data information of the first frame image, so as to ensure that the image data information processed by the second processor for depth information processing matches the first frame image received by the first processor, thereby improving the accuracy of the obtained depth information.

[0082] In the above image processing method, the second processor obtains the image data information of the first frame image received by the first processor, processes the depth information based on the image data information of the first frame image, updates the obtained depth information of the first frame image to the metadata of the second frame image based on the reception of the second frame image by the first processor and before the end of the reception of the second frame image by the first processor, and the metadata of the second frame image instructs the receiver to process the corresponding first frame image based on the depth information of the first frame image in the metadata of the second frame image after being sent to the receiver by the first processor. In the image processing process, the image data information processed by the second processor for depth information processing is the image data information of the image received by the first processor, which ensures the consistency of the image data information of the same image in different processing processes, thereby improving the accuracy of the obtained depth information.

[0083] In one embodiment, obtaining the image data information of the first frame image includes triggering a frame end interrupt event based on the reception of the first frame image by the first processor to obtain the image data information of the first frame image.

[0084] The frame end interrupt event refers to an interrupt event of the first processor ending the image reception. After the first processor receives all the image data of the first frame image, the photosensitive element of the camera needs to go through a vertical blanking period to adjust to process the second frame image, and the first processor generates a frame end interrupt event of the first frame image to trigger the second processor to process the depth information of the first frame image.

[0085] Specifically, when the first processor receives the first frame image, a frame end interrupt event is triggered, and the second processor acquires the image data information of the first frame image in response to the frame end interrupt event. In a specific application, when the frame end interrupt event is triggered, the first processor can directly send a trigger signal to the second processor to trigger the second processor to acquire the image data information of the first frame image for depth information processing. In another application, when the frame end interrupt event is triggered, the scheduling processor can send a trigger signal to the second processor in response to the frame end interrupt event to trigger the second processor to acquire the image data information of the first frame image for depth information processing. In a specific implementation, the trigger signal sent by the first processor or the scheduling processor to the second processor can carry the image data information of the first frame image, or can carry storage location information of the image data information of the first frame image, so that the second processor acquires the image data information of the first frame image according to the storage location information.

[0086] In the embodiment, based on the first processor triggering the frame end interrupt event to receive the first frame image, the second processor acquires the image data information of the first frame image for depth information processing, which can ensure that the first frame image has ended receiving and the corresponding image data will not be changed, and can ensure that the image data information for depth information processing by the second processor matches the first frame image received by the first processor, thereby facilitating to improve the accuracy of the obtained depth information.

[0087] In one embodiment, the depth information of the first frame image is obtained based on the image data information of the first frame image for depth information processing, including: depth estimation based on the image data information of the first frame image to obtain the depth information of the first frame image before the first processor triggers a frame start interrupt event to receive the second frame image.

[0088] The frame start interrupt event refers to an interrupt event when the first processor starts to receive the image. The light sensing element of the camera undergoes a vertical blanking period adjustment to process the second frame image, and starts to output the image data of the second frame image to the first processor. When the first processor starts to receive the second frame image, the frame start interrupt event is triggered.

[0089] Specifically, the second processor performs depth estimation based on the image data information of the first frame image, and specifically can perform depth estimation on the image data information through a pre-trained artificial neural network model to obtain depth information of the first frame image. The second processor performs depth estimation on the image data information, and the depth estimation is completed before the first processor triggers the frame start interrupt event of receiving the second frame image. That is, the second processor completes the depth estimation processing of the first frame image before the first processor receives the second frame image, so that the first processor and the second processor can process different frame images, and the isolation of the processing of the first processor and the second processor is realized. That is, when the first processor receives the second frame image, the depth estimation processing of the first frame image by the second processor is not affected, and the control of the first processor and the second processor can be simplified.

[0090] In the embodiment, the second processor completes the depth information processing of the first frame image before the first processor receives the second frame image, that is, before the first processor triggers the frame start interrupt event, to obtain the depth information of the first frame image. Therefore, the first processor and the second processor can process different frame images, the isolation of the processing of the first processor and the second processor is realized, and it is ensured that the image data information processed by the second processor for depth information processing matches the first frame image received by the first processor, thereby improving the accuracy of the obtained depth information.

[0091] In one embodiment, based on receiving the second frame image by the first processor, and before the first processor receives the second frame image, the depth information of the first frame image is updated into the metadata of the second frame image, including: based on the first processor triggering the frame start interrupt event of receiving the second frame image, and before the first processor triggering the frame end interrupt event of receiving the second frame image, the depth information of the first frame image is updated into the metadata of the second frame image.

[0092] Wherein, the frame start interrupt event refers to the interrupt event of the first processor starting to receive the image, and the frame end interrupt event refers to the interrupt event of the first processor ending to receive the image.

[0093] Specifically, when the first processor triggers the frame start interrupt event of receiving the second frame image, it indicates that the first processor starts to receive the second frame image, and the second processor updates the depth information of the first frame image into the metadata of the second frame image. Before the first processor triggers the frame end interrupt event of receiving the second frame image, that is, before the first processor receives the second frame image, the processing of updating the depth information of the first frame image into the metadata of the second frame image is completed, so that when the first processor sends the metadata of the second frame image to the receiving end, the depth information of the first frame image can also be sent together, thereby ensuring the processing efficiency of the metadata of the second frame image.

[0094] In this embodiment, the frame start interrupt event of receiving the second frame image is triggered by the first processor, the depth information of the first frame image is updated to the metadata of the second frame image by the second processor, and the processing of updating the depth information of the first frame image to the metadata of the second frame image is completed before the first processor triggers the frame end interrupt event of receiving the second frame image, so that when the first processor completes the reception of the second frame image, the first processor can send the metadata of the second frame image carrying the depth information of the first frame image to the receiving end, and the processing efficiency of sending the metadata of the second frame image to the sending end is determined.

[0095] In one embodiment, the first processor includes an image processor, and the second processor includes a neural network processor.

[0096] The image processor (ISP, Image Signal Processor) is used for post-processing of signals output by a front-end image sensor, and the main functions include linear correction, noise removal, bad point removal, interpolation, white balance, automatic exposure control, etc., and the ISP can restore the scene details well under different optical conditions. The neural network processor (NPU, Neural-network Processing Unit) can run a deep learning algorithm to process depth information for images.

[0097] Specifically, the first processor can be an image processor to receive image data transmitted by a photosensitive element through the image processor and process the received image data. The second processor can be a neural network processor to process depth information for the image received by the first processor to obtain depth information corresponding to the image. The frame start interrupt event and the frame end interrupt event generated by the image processor in the process of receiving the image transmitted by the photosensitive element can simplify the control of the neural network processor for depth information processing and improve the efficiency of depth information processing.

[0098] In this embodiment, the frame start interrupt event and the frame end interrupt event generated by the image processor in the process of receiving the image transmitted by the photosensitive element are used as trigger signals to trigger the neural network processor to process depth information, which simplifies the control of the neural network processor for depth information processing and is beneficial to improve the efficiency of depth information processing.

[0099] In one embodiment, the image data information of the first frame image is obtained by: determining the state of the neural network processor based on the reception of the first frame image by the image processor; and obtaining the image data information of the first frame image based on the neural network processor being in a normal working state.

[0100] Specifically, when the receiving of the first frame image by the image processor ends, i.e., a frame end interrupt event is triggered based on the receiving of the first frame image by the image processor, the state of the neural network processor is determined to determine whether the neural network processor can implement the depth information processing of the first frame image, such as detecting whether the neural network processor fails, whether the computing resources are sufficient, etc. If it is determined that the neural network processor is in a normal working state, i.e., the neural network processor can implement the depth information processing of the first frame image, the image data information of the first frame image is acquired by the neural network processor to perform the depth information processing on the first frame image. In a specific application, if the neural network processor is in an abnormal state, indicating that the neural network processor cannot perform the depth information processing on the first frame image, it can be prompted to troubleshoot the neural network processor. In the case where the neural network processor returns to the normal working state, the image data information of the first frame image is acquired by the neural network processor to perform the depth information processing.

[0101] In this embodiment, based on the determination that the neural network processor is in a normal working state, the image data information of the first frame image is acquired by the neural network processor to perform the depth information processing, which can ensure that the neural network processor can support the depth information processing and ensure the normal operation of the depth information processing.

[0102] In one embodiment, the image data information of the first frame image includes image attribute information and data storage location information of the first frame image. As shown in Figure 3 As shown, the steps of the depth information processing, i.e., the depth information processing based on the image data information of the first frame image, include:

[0103] In step 302, the image data of the first frame image is acquired according to the data storage location information.

[0104] The image data information of the first frame image includes image attribute information and data storage location information of the first frame image. The image attribute information of the first frame image is description information describing the first frame image, such as the height, width or bit width of the first frame image, etc. The data storage location information refers to the location where the image data of the first frame image is stored. According to the data storage location information, the specific image data of the first frame image can be obtained. The image data refers to the data of each pixel point in the first frame image, i.e., the first frame image is composed of image data.

[0105] Specifically, the image data information obtained by the second processor includes image attribute information and data storage location information of the first frame image. The second processor acquires the image data of the first frame image from the corresponding storage location according to the data storage location information.

[0106] ​At step 304, depth estimation is performed according to the image data and the image attribute information of the first frame image to obtain the depth information of the first frame image.

[0107] Specifically, the second processor performs depth estimation according to the image data and the image attribute information of the first frame image. For example, image features of the first frame image can be constructed according to the image data and the image attribute information of the first frame image, and the image features are input into a pre-trained depth estimation model to perform depth estimation based on the image features by the depth estimation model, and the depth information of the first frame image is output by the depth estimation model. The depth estimation model can be obtained by training training sample images carrying depth information labels. The depth information labels can be depth images corresponding to the training sample images. The obtained depth estimation model can perform depth estimation according to the input image features and output corresponding depth images.

[0108] In this embodiment, the second processor performs depth estimation based on the image data and the image attribute information of the first frame image, which can be based on multi-dimensional information of the first frame image to improve the accuracy of the obtained depth information.

[0109] In one embodiment, based on receiving the second frame image by the first processor, and before the first processor finishes receiving the second frame image, the depth information of the first frame image is updated into the metadata of the second frame image, including: based on receiving the second frame image by the first processor, determining the metadata of the second frame image; before the first processor finishes receiving the second frame image, adding the depth information of the first frame image into the metadata of the second frame image.

[0110] The metadata of the second frame image is data describing the second frame image, which can be used as attribute information of the second frame image. Different fields can be set in the metadata of the second frame image, and different types of attribute information can be written in different fields. Each field in the metadata can be flexibly configured according to actual needs, such as setting the length, position, data type, corresponding attribute information type, etc. of the field. The metadata can be obtained by the first processor when receiving the second frame image. For example, the first processor analyzes the image data of the second frame image to obtain the attribute information of the second frame image, and obtains the metadata of the second frame image based on the attribute information.

[0111] Specifically, when determining that the first processor receives the second frame image, the second processor can determine the metadata of the second frame image, and specifically can determine the storage location of the metadata of the second frame image, and query the metadata of the second frame image according to the storage location. The second processor adds the depth information of the first frame image to the metadata of the second frame image, and completes the process of adding the depth information of the first frame image to the metadata of the second frame image before the first processor finishes receiving the second frame image. That is, the second processor completes the process of adding the depth information of the first frame image to the metadata of the second frame image before the first processor finishes receiving the second frame image, so that the first processor can timely send the depth information of the first frame image following the metadata of the second frame image to the receiving end after receiving the second frame image, so as to ensure the efficiency of processing the first frame image.

[0112] In the embodiment, when determining that the first processor receives the second frame image, the second processor determines the metadata of the second frame image, adds the depth information of the first frame image to the metadata of the second frame image, and completes the process of adding the depth information of the first frame image to the metadata of the second frame image before the first processor finishes receiving the second frame image, so that the first processor can timely send the depth information of the first frame image following the metadata of the second frame image to the receiving end after receiving the second frame image, and ensure the efficiency of processing the first frame image.

[0113] In one embodiment, before the first processor finishes receiving the second frame image, the depth information of the first frame image is added to the metadata of the second frame image, including: determining a depth information field from the metadata of the second frame image; and writing the depth information of the first frame image into the depth information field before the first processor finishes receiving the second frame image.

[0114] In the embodiment, the metadata of the second frame image includes a depth information field for storing depth information, and specifically for storing the depth information of the first frame image. The second processor can add the depth information of the first frame image to the metadata of the second frame image by writing the depth information of the first frame image into the depth information field of the metadata of the second frame image.

[0115] Specifically, the second processor further determines the depth information field in the metadata of the second frame image, and specifically can determine the depth information field from the metadata of the second frame image according to the identification of each field in the metadata. The second processor writes the depth information of the first frame image into the depth information field, and completes the process of writing the depth information of the first frame image into the depth information field before the first processor finishes receiving the second frame image.

[0116] In this embodiment, the second processor adds the depth information of the first frame image to the metadata of the second frame image by writing the depth information of the previous frame image into the depth information field of the metadata of the second frame image, so that the first processor can timely send the depth information of the first frame image following the metadata of the second frame image to the receiving end after receiving the second frame image, and the efficiency of processing the first frame image is ensured.

[0117] In one embodiment, as shown in Figure 4 , an image processing method is provided, which is applied to a scheduling processor in Figure 1 , and the scheduling processor can be a processor in an electronic device or a server. In this embodiment, the method comprises the following steps:

[0118] Step 402, receiving a first frame image by a first processor.

[0119] The first frame image is received by the first processor, and specifically, the first processor can receive the first frame image from a photosensitive element of a camera. When the camera is shooting, the photosensitive element of the camera sends the shot image to the first processor for receiving.

[0120] Specifically, the scheduling processor can trigger the first processor to perform image receiving tasks, that is, control the first processor to receive the first frame image from the photosensitive element of the camera. The first frame image is the image currently received by the first processor from the photosensitive element of the camera.

[0121] Step 404, obtaining image data information of the first frame image by a second processor, so that the second processor performs depth information processing based on the image data information of the first frame image, and updates the obtained depth information of the first frame image to the metadata of the second frame image based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image.

[0122] The image data information refers to information related to the first frame image, which specifically can include attribute information, storage location information, etc. of the first frame image, and can also include specific image data of the first frame image. The depth information can include depth data of the image, and specifically can include a depth image, which is an image taking the distance from the image collector to each point in the scene, i.e. the depth, as the pixel value, and directly reflects the geometric shape of the visible surface of the scene. The metadata of the image is data describing the image, which can be used as attribute information of the image.

[0123] Specifically, the scheduling processor can schedule the second processor to obtain the image data information of the first frame image when the first processor finishes receiving the first frame image, i.e., the first processor completes the receiving process of the first frame image. The image data information can include attribute information and storage location information of the first frame image. The second processor can obtain the image data of the first frame image based on the storage location information and perform depth estimation based on the image data and the attribute information of the first frame image to obtain the depth information of the first frame image. In a specific implementation, the second processor can perform depth calculation on the image data information of the first frame image based on a deep learning algorithm to obtain the depth information of the first frame image, such as depth data of the first frame image. When the scheduling processor determines that the first processor receives the second frame image, the scheduling processor can update the depth information of the first frame image to the metadata of the second frame image through the second processor. Specifically, the scheduling processor can determine the storage location of the metadata of the second frame image, update the depth information of the first frame image to the storage location of the metadata of the second frame image through the second processor, and complete the updating process of the depth information of the first frame image to the metadata of the second frame image before the first processor finishes receiving the second frame image.

[0124] In step 406, the first processor sends the metadata of the second frame image to the receiving end to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0125] Specifically, for the metadata of the second frame image, the scheduling processor can send it to the receiving end through the first processor, so that the receiving end obtains the depth information of the first frame image from the metadata of the second frame image after obtaining the metadata of the second frame image, and processes the corresponding first frame image according to the depth information of the first frame image, such as image segmentation, image blurring, etc. The first frame image can also be sent to the receiving end by the first processor. Specifically, the first processor can send the received first frame image to the receiving end after completing the receiving process of the first frame image under the control of the scheduling processor.

[0126] In one specific application, for example, Figure 5As shown, the scheduling processor controls the first processor to perform image receiving, and the first processor receives a first frame image. The scheduling processor controls the second processor to perform depth information processing on the first frame image after determining that the first processor completes the receiving of the first frame image, i.e., when the first processor ends the receiving of the first frame image. Specifically, the second processor acquires image data information of the first frame image, and performs depth estimation based on the image data information of the first frame image to obtain depth information of the first frame image. The depth information of the first frame image can specifically include depth data of the first frame image, and can also include attribute data of the depth data. When the first processor triggers the receiving of a second frame image, the scheduling controller controls the second processor to perform update processing on the depth information of the first frame image. Specifically, the second processor updates the depth information of the first frame image to metadata of the second frame image, and completes the update processing on the depth information of the first frame image, i.e., completes the processing of updating the depth information of the first frame image to the metadata of the second frame image, before the first processor ends the receiving of the second frame image. The scheduling processor controls the first processor to send the metadata of the second frame image, and the first processor sends the metadata of the second frame image to a receiving end. The receiving end can extract the depth information of the first frame image from the metadata of the second frame image, and perform processing, such as three-dimensional reconstruction, virtualization processing, etc., according to the depth information of the first frame image and the corresponding first frame image.

[0127] In this specific application, the scheduling processor controls the second processor to perform depth information processing on the first frame image, including calculating the depth information of the first frame image, and updating the calculated depth information of the first frame image to the metadata of the second frame image. Both of the above processes are performed between the end of the receiving of the first frame image by the first processor and the end of the receiving of the second frame image, i.e., after the first processor completes the receiving of the first frame image and will not change the image data information of the first frame image, so that the image data information processed by the second processor for depth information processing matches the first frame image received by the first processor, thereby improving the accuracy of the obtained depth information.

[0128] In the image processing method, the second processor acquires image data information of the first frame image received by the first processor, performs depth information processing based on the image data information of the first frame image, and updates the obtained depth information of the first frame image into metadata of a second frame image based on the fact that the second frame image is received by the first processor and before the end of the reception of the second frame image by the first processor. The first processor sends the metadata of the second frame image to the receiving end to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image. In the image processing process, the second processor processes image data information each time, which is the image data information of the image received by the first processor. The consistency of the image data information of the same image in different processing processes is ensured, thereby improving the accuracy of the obtained depth information and improving the image processing effect when processing based on the depth information.

[0129] In one embodiment, the image processing method further comprises: determining a processing statistical duration of the depth information processing of the second processor on the received image; determining a receiving interval duration between adjacent two frame images received by the first processor based on the processing statistical duration; and the receiving interval duration is greater than or equal to the processing statistical duration.

[0130] The second processor is configured to perform depth information processing on the received image, which can include depth estimation of the received image. The second processor can also be configured to statistically determine the duration required for depth estimation of each frame image. In addition, the second processor can be configured to statistically determine the historical time consumption of depth estimation, thereby obtaining the processing statistical duration. The processing statistical duration represents the time consumption of the second processor when performing depth information processing based on the image data information of the image. The length of the processing statistical duration is related to the computing power of the second processor and the data size of the processed image data information. The receiving interval duration refers to the interval duration between the end of the reception of the first frame image by the first processor and the start of the reception of the second frame image by the first processor. In other words, the receiving interval duration is the interval duration between adjacent two frame images received by the first processor. During this interval duration, the first processor does not perform image reception.

[0131] Specifically, the scheduling processor can determine a processing statistical duration of the second processor for depth information processing on the received image. In a specific application, the scheduling processor can statistically determine the time consumption of the second processor for depth information processing on the historically received image, and obtain the processing statistical duration based on the statistical result, which is used to represent the time consumption of the second processor for depth information processing based on the image data of the image. For example, the scheduling processor can obtain the processing statistical duration by weightedly averaging the time consumption of the second processor for depth information processing on the historically received image. The scheduling processor determines the receiving interval duration of the first processor based on the processing statistical duration of the second processor, and the receiving interval duration is greater than or equal to the processing statistical duration. In a specific application, the scheduling processor can set the receiving interval duration of the first processor based on the processing statistical duration of the second processor, so that the receiving interval duration of the first processor is not less than the processing statistical duration of the second processor, thereby enabling the second processor to have sufficient time to complete the depth information processing during the interval between the reception of adjacent two frames of images by the first processor, and obtain the depth information of the image.

[0132] In the embodiment, the scheduling processor determines the receiving interval duration between adjacent two frames of images of the first processor based on the processing statistical duration of the second processor, so that the receiving interval duration is greater than or equal to the processing statistical duration, thereby enabling the second processor to have sufficient time to complete the depth information processing during the interval between the reception of adjacent two frames of images by the first processor, and obtain the depth information of the image.

[0133] In one embodiment, the image processing method further comprises: determining a virtualization processing parameter based on the image data and the depth information of the first frame of image; and performing image virtualization processing on the first frame of image by using the virtualization processing parameter.

[0134] The virtualization processing parameter is a parameter for virtualization processing of the image, and the image can be virtualized by using the virtualization processing parameter to obtain a virtualized image. Specifically, the scheduling processor obtains the image data of the first frame of image, and determines the virtualization processing parameter based on the image data and the depth information of the first frame of image. For example, the scheduling processor can segment the first frame of image according to the depth information and the image data, determine the front-back relationship of the scene in the first frame of image, and generate a corresponding virtualization processing parameter to control the virtualization degree of the first frame of image. The scheduling processor can perform image virtualization processing on the first frame of image based on the obtained virtualization processing parameter to obtain a virtualized image corresponding to the first frame of image, thereby realizing the virtualization processing of the first frame of image. The virtualization of the first frame of image by using the depth information can ensure the virtualization effect of the image and make the virtualized image more natural.

[0135] In the embodiment, the scheduling processor determines the virtualization processing parameter according to the image data and the depth information of the first frame image, and performs image virtualization processing on the first frame image based on the virtualization processing parameter, so that the virtualization can be performed based on the accurate depth information of the first frame image, the image virtualization effect can be ensured, and the virtualized image is more natural.

[0136] In one embodiment, the image processing method further includes: based on the end of receiving the first frame image by the first processor, sending, by the first processor, the image data of the first frame image to the receiving end, to instruct the receiving end to process the image data corresponding to the first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0137] The image data of the first frame image is data obtained by the first processor in the image receiving process. The image data of the first frame image is sent to the receiving end, so that the receiving end can process the image data according to the image data and the depth information of the first frame image, such as image virtualization processing and three-dimensional reconstruction processing.

[0138] Specifically, the scheduling processor determines that the first processor has completed the reception of the first frame image when the first processor triggers the frame end interrupt event of receiving the first frame image. The scheduling processor sends the image data of the first frame image to the receiving end through the first processor, so that the receiving end processes the image data corresponding to the first frame image according to the depth information of the first frame image in the metadata of the second frame image. The image data of the first frame image is sent to the receiving end when the first processor ends the reception of the first frame image, and the depth information of the first frame image is sent to the receiving end along with the metadata of the second frame image, that is, the image data and the depth information of the first frame image received by the receiving end are different by one frame image, which can ensure the accuracy of the depth information obtained by the receiving end and is beneficial to improve the processing effect of the first frame image based on the depth information.

[0139] In the embodiment, the scheduling processor sends the image data of the first frame image to the receiving end through the first processor when the first processor ends the reception of the first frame image, to instruct the receiving end to process the image data corresponding to the first frame image according to the depth information of the first frame image in the metadata of the second frame image, so that the image data and the depth information of the first frame image received by the receiving end are different by one frame image, which can ensure the accuracy of the depth information obtained by the receiving end and is beneficial to improve the processing effect of the first frame image based on the depth information.

[0140] In one embodiment, the depth information comprises a depth image of the first frame image; the image processing method further comprises: obtaining attribute information corresponding to the depth image; and updating, by the first processor, the attribute information into metadata of the second frame image based on receiving, by the first processor, the second frame image and before the first processor finishes receiving the second frame image.

[0141] The depth image refers to an image in which distances from an image collector to each point in a scene, i.e., depths, are taken as pixel values, and directly reflects geometrical shapes of visible surfaces of a scene. The attribute information corresponding to the depth image refers to information describing the depth image, such as various information including width, height, or bit width of the depth image.

[0142] Specifically, the depth information comprises a depth image of the first frame image, and the scheduling processor obtains attribute information corresponding to the depth image. The attribute information corresponding to the depth image can be generated by the second processor when processing the depth information, or can be obtained by the first processor or the scheduling processor based on the depth image analysis. The scheduling processor updates, by the first processor, the attribute information corresponding to the depth image into metadata of the second frame image based on receiving, by the first processor, the second frame image, and finishes the updating of the attribute information into the metadata of the second frame image before the first processor finishes receiving the second frame image.

[0143] Further, the metadata of the second frame image is sent to the receiving end by the first processor to instruct the receiving end to process the corresponding first frame image based on the depth information of the first frame image and the attribute information in the metadata of the second frame image.

[0144] Specifically, the scheduling processor sends, by the first processor, the metadata of the second frame image to the receiving end, and the receiving end extracts the depth image of the first frame image and the attribute information from the metadata of the second frame image after receiving the metadata of the second frame image, and thus processes the corresponding first frame image based on the depth image of the first frame image and the attribute information.

[0145] In this embodiment, the scheduling processor updates the attribute information corresponding to the depth image of the first frame image obtained by the first processor into the metadata of the second frame image, and after the first processor sends the metadata of the second frame image to the receiving end, the receiving end can process the corresponding first frame image according to the depth image and attribute information of the first frame image, and can use the attribute information corresponding to the depth image for image processing, which is beneficial to further improve the image processing effect.

[0146] The application also provides an application scenario for applying the image processing method. Specifically, the image processing method is applied in the application scenario as follows:

[0147] When performing three-dimensional reconstruction on the image obtained by the camera shooting, the depth information of the image needs to be obtained, such as obtaining the depth image corresponding to the image, so as to perform three-dimensional reconstruction according to the depth image. When the camera shoots the image, the image data is sent to the first processor through the photosensitive element, the first processor receives the image data of the first frame image, when the first processor ends the reception of the first frame image, the second processor obtains the image data information of the first frame image, and performs depth information processing based on the image data information of the first frame image to obtain the depth information of the first frame image. In the case that the first processor continues to receive the second frame image, the second processor writes the depth information of the first frame image into the metadata of the second frame image, and completes the processing of writing the depth information of the first frame image into the metadata of the second frame image before the first processor ends the reception of the second frame image. The first processor can send the metadata of the second frame image to the receiving end to instruct the receiving end to extract the depth information of the first frame image from the metadata of the second frame image, and perform three-dimensional reconstruction processing according to the depth information of the first frame image and the corresponding first frame image.

[0148] In one embodiment, as shown in Figure 6 , an image processing method is provided, which is applied to Figure 1 the first processor in the above method. The scheduling processor can be a processor in an electronic device or a server. In this embodiment, the method comprises the following steps:

[0149] Step 602, receiving a first frame image and sending the first frame image to a receiving end.

[0150] Wherein, the first frame image is received by the first processor, which can be received by the first processor from the photosensitive element of the camera. When the camera shoots, the photosensitive element of the camera sends the shot image to the first processor, which is received by the first processor. The receiving end is used for processing the image, such as image segmentation, image blurring or three-dimensional reconstruction and various processing.

[0151] Specifically, the first processor performs an image receiving task, that is, the first processor receives a first frame image from a photosensitive element of the camera. The first frame image is an image currently received by the first processor from the photosensitive element of the camera. The first processor sends the received first frame image to a receiving end. Specifically, after the reception of the first frame image is completed, the first processor sends the first frame image to the receiving end.

[0152] In step 604, based on receiving the second frame image, metadata of the second frame image is sent to the receiving end to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image; wherein the depth information of the first frame image is obtained by the second processor based on image data information of the first frame image, and the second processor updates the obtained depth information of the first frame image into the metadata of the second frame image based on receiving the second frame image and before the end of receiving the second frame image.

[0153] The metadata of the image is data describing the image, which can be used as attribute information of the image. The metadata of the image can be obtained by the first processor when receiving the image. The image data information refers to information related to the first frame image, which can specifically include attribute information, storage location information, etc. of the first frame image, and can also include specific image data of the first frame image. The depth information can include depth data of the image, which can specifically include a depth image. The depth image refers to an image in which distances from an image collector to each point in a scene, that is, depths, are used as pixel values, which directly reflects the geometric shape of the visible surface of the scene.

[0154] Specifically, for the metadata of the second frame image, the first processor sends it to the receiving end, so that the receiving end obtains the depth information of the first frame image from the metadata of the second frame image after obtaining the metadata of the second frame image, and processes the corresponding first frame image according to the depth information of the first frame image. For the depth information of the first frame image, the second processor can obtain the image data information of the first frame image when the first processor ends receiving the first frame image, that is, the first processor completes the receiving processing of the first frame image. The image data information can include attribute information and storage location information of the first frame image. The second processor can obtain the image data of the first frame image based on the storage location information, and perform depth estimation based on the image data and the attribute information of the first frame image to obtain the depth information of the first frame image. In a specific implementation, the second processor can perform depth calculation on the image data information of the first frame image based on a deep learning algorithm to obtain the depth information of the first frame image, such as depth data of the first frame image. When it is determined that the first processor receives the second frame image, the second processor can update the depth information of the first frame image to the metadata of the second frame image. Specifically, the storage location of the metadata of the second frame image can be determined, the depth information of the first frame image is updated to the storage location of the metadata of the second frame image by the second processor, and the processing of updating the depth information of the first frame image to the metadata of the second frame image is completed before the first processor ends receiving the second frame image.

[0155] The above image processing method includes receiving, by a first processor, a first frame image and sending the first frame image to a receiving end, sending, based on receiving a second frame image, metadata of the second frame image to the receiving end to instruct the receiving end to process a corresponding first frame image according to depth information of the first frame image in the metadata of the second frame image. The depth information of the first frame image is obtained by a second processor based on image data information of the first frame image, and the second processor updates the obtained depth information of the first frame image to the metadata of the second frame image based on receiving the second frame image and before ending receiving the second frame image. In the image processing process, the second processor processes the image data information each time, which is the image data information of the image that has been completely received by the first processor, ensuring the consistency of the image data information of the same image in different processing processes, thereby improving the accuracy of the obtained depth information and improving the image processing effect when processing based on the depth information.

[0156] The application also provides an application scenario applying the above image processing method. Specifically, the image processing method is applied in the application scenario as follows:

[0157] As Figure 7As shown, for the blurring of the image, the farther away from the focal plane, the larger the circle of confusion, and the more blurred the image. That is, the closer to the focal plane, the clearer the image. The ideal effect of blurring is that although everything outside the depth of field is blurred, the degree of blurring is different. The closer to the focal plane, the clearer the image, and the farther away from the focal plane, the more blurred the image.

[0158] At present, the commonly used image blurring schemes include two kinds based on one camera and two cameras. Among them, the image blurring based on one camera is mostly achieved by blurring algorithm processing on the area outside the focusing area of the shooting object; but due to the use of only one camera, it is difficult to obtain the depth information of the lens and the shooting area, resulting in similar blurring degree of different areas when algorithm processing. Deep learning can also be used for background segmentation, and a neural network is trained using a data set to achieve foreground and background segmentation. After segmentation, it is determined which objects are the main objects that we want to focus clearly, and which objects are the background that need to be blurred. However, if the blurring is only based on the foreground and background segmentation results, the blurred photo is not natural. The main reason is still the lack of depth data, which cannot be used for distance-dependent blurring, and the segmentation algorithm is not perfect, and the foreground and background may be misjudged. Based on two cameras for blurring, a depth image (Depth Map) is constructed using dual cameras to infer the front and back relationship of the scene and control the degree of blurring. Dual cameras can calculate the distance of each pixel point from the focal plane to calculate the blurring according to the focal plane distance.

[0159] At present, the commonly used processing method for single camera depth calculation is to process by image processor ISP and neural network processor NPU at the same time, specifically by ISP for image processing, and NPU for running depth learning algorithm to calculate the depth information of the image. Such a need for synchronization of ISP and NPU two hardware and software coordination. Especially when the image data output by the photosensitive element Sensor changes, such as DOL (Digital overlay wide dynamic range, digital overlay wide dynamic technology) switching, resolution change, etc., it will make the control difficulty become more complex, and it is easy to cause the ISP and NPU processing data mismatch, resulting in inaccurate calculation results, which greatly affects the processing effect of the subsequent image.

[0160] Based on this, in order to reduce the complexity of the scheduling of the software and hardware work, the image processing method provided in the embodiment utilizes the SOF (Start Of Frame, frame start) and EOF (End Of Frame, frame end) interrupts generated by the ISP processing of the image, specifically utilizes the EOF of the image transmission to trigger the NPU to perform deep calculation, fills the depth information into the metadata at the SOF, to control the NPU to calculate the depth image, so as to simplify the processing process of the depth image, and make the entire depth image processing process simple and reliable. The image processing method provided in the embodiment synchronously calculates the depth information while the image is being transmitted by using a specific hardware architecture, accelerates the generation process of the depth information, and uses a heterogeneous logical control processing mode, so that the entire process is efficient and stable.

[0161] Specifically, the image processing method provided in the embodiment is based on scheduling a processor, such as the hardware architecture of the Explorer chip and the time setting of the frame output of the photosensitive element Sensor, to complete the depth information calculation function. For the AON (Always On, always online) sensor, that is, the photosensitive element that is always online, the image data output is continuously performed, and the sensor is generally applied in a front-looking scene. In the Explorer chip, the ISP and the NPU are arranged, the ISP can perform regular processing on the image, and the NPU can perform deep learning calculation. The two HWs (Hardware) can run simultaneously and do not affect each other. Based on the above hardware architecture, in this scene, the Explorer chip can perform shunt processing on the image data, that is, the image is transmitted to the ISP for traditional image processing, and on the other hand, the image is transmitted to the NPU to perform deep learning and calculate the subsequent required depth information.

[0162] For example, the Explorer chip can be used in a front-looking camera, and the AON sensor is used as the photosensitive element of the camera. The AON sensor is always online, and the image data output is continuously performed. The Explorer chip can perform shunt processing on the image data, that is, the image is transmitted to the ISP for traditional image processing, and on the other hand, the image is transmitted to the NPU to perform deep learning and calculate the subsequent required depth information. Figure 8As shown, the time length for the image processor ISP to receive each frame of image from the photosensitive element Sensor is 21 ms (milliseconds), the time interval between two adjacent frames is 12 ms, and the time consumed by the neural network processor NPU to calculate the depth information of one frame of image is 9 ms. When the image processor ISP and the neural network processor NPU are triggered to start working, at the time when the image processor ISP ends receiving the 0th frame of image, the neural network processor NPU is triggered to start calculating the depth information of the 0th frame of image, and the depth data of the 0th frame of image is obtained after 9 ms, which can specifically include the depth image of the 0th frame of image. After 12 ms from the time when the image processor ISP ends receiving the 0th frame of image, the image processor ISP is triggered to start receiving the 1st frame of image, and the neural network processor NPU is triggered to write the obtained depth data into the metadata of the 1st frame of image, so that the metadata of the 1st frame of image includes the metadata information of the 1st frame of image and the depth data of the 0th frame of image. In addition, the image processor ISP also updates the attribute information of the depth data into the metadata of the 1st frame of image. The image processor ISP can send the metadata of the 1st frame of image to the receiving end, and the receiving end can extract the depth data of the 0th frame of image and the attribute information of the depth data from the metadata to process the 0th frame of image, specifically, to perform the blurring processing.

[0163] In this way, the depth information is processed by the image processor ISP and the neural network processor NPU, when the image processor ISP ends receiving the Nth frame of image, the neural network processor NPU is triggered to start calculating the depth information of the Nth frame of image, and the depth data of the Nth frame of image is obtained after 9 ms, which can specifically include the depth image of the Nth frame of image. After 12 ms from the time when the image processor ISP ends receiving the Nth frame of image, the image processor ISP is triggered to start receiving the N+1th frame of image, and the neural network processor NPU is triggered to write the obtained depth data into the metadata of the N+1th frame of image, so that the metadata of the N+1th frame of image includes the metadata information of the N+1th frame of image and the depth data of the Nth frame of image. In addition, the image processor ISP also updates the attribute information of the depth data of the Nth frame of image into the metadata of the N+1th frame of image. The image processor ISP can send the metadata of the N+1th frame of image to the receiving end, and the receiving end can extract the depth data of the Nth frame of image and the attribute information of the depth data from the metadata to process the Nth frame of image.

[0164] Furthermore, to enable the ISP and NPU to work collaboratively in this scenario, the sensor needs to be configured. In this scenario, the SOF and EOF intervals for outputting one frame of image data from the sensor are required to be 21ms, and the vblk (vertical blanking) interval 12ms. This provides the NPU with sufficient time for depth calculation. Here, vblk refers to the time interval between the end of one frame read and the start of the next frame read, i.e., the interval between the ISP receiving two adjacent image frames.

[0165] Specifically, when the ISP receives the EOF (End of Frame) of the first image, it triggers the NPU to start working and informs the NPU of the data information and storage location of the first image, including width, height, and bit width. After receiving the data information and storage location of the first image, the NPU begins to calculate the corresponding depth data for that first image. The estimated time for the NPU to calculate the information for one image frame is 9ms, so it will be completed before the SOF (Send of Frame) of the second frame. After processing, the NPU stores the depth data in a designated buffer and notifies the Explorer chip of relevant attribute information, which describes the depth data. When the SOF of the second image is triggered, the Explorer chip updates the metadata Info of the second image with the depth data and corresponding attribute information of the first image, and the ISP sends the metadata of the second image to the receiving end. The receiving end will obtain the depth data and corresponding attribute information of the first frame image from the metadata Info of the second frame image, and can further process the first frame image based on the depth data and corresponding attribute information, such as performing blurring processing.

[0166] like Figure 9 As shown, during the depth information processing by the Image Processor (ISP) and the Neural Processing Unit (NPU), the ISP detects the state of the NPU when it triggers an image frame end interrupt for the first frame. If the NPU is in normal working condition, the Explorer chip triggers the NPU to perform depth calculations until the calculations are complete and feedback is provided. When the ISP triggers an image frame start interrupt for the second frame, it adds the attribute information corresponding to the depth data of the first frame to the metadata of the second frame. When the image frame end interrupt for the second frame is triggered, the ISP sends the metadata of the second frame to the receiving end. This achieves the transmission of depth information, including the depth data and corresponding attribute information of the first frame, for further processing by the receiving end.

[0167] In a specific application, the logical relationship between the processing flows of the ISP and the NPU can be confirmed by checking the log. Moreover, the image data and the depth information output are different by one frame, and whether the image processing method provided in the embodiment is adopted can be determined by the order in which the image data and the depth information are received.

[0168] The image processing method provided in the embodiment uses the hardware characteristics of the Explorer chip and the image transmission flow to complete the use scenario in which the ISP and the NPU work together, can maximize the use of the Explorer chip hardware and efficiently use data transmission, can realize a relatively complex scene requirement by using a lower hardware architecture and a simpler control scheme. The use of heterogeneous control logic to coordinate the functions of the ISP and the NPU can greatly simplify the complexity of the control logic and enhance the stability of the control logic. The image processing method provided in the embodiment combines the Explorer chip hardware architecture, the specific output of the light sensing element Sensor, and the characteristics of image transmission through scheduling control, and cooperates to realize the processing of the Explorer chip image and the output of the depth information corresponding to the image at the same time.

[0169] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0170] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, and therefore the specific limitations in one or more image processing device embodiments provided below can refer to the limitations of the image processing method described above, which will not be repeated here.

[0171] In one embodiment, as shown in Figure 10 An image processing device 1000 is provided, including an image data information acquisition module 1002, a depth information processing module 1004, and a depth information updating module 1006, wherein:

[0172] The image data information obtaining module 1002 is configured to obtain image data information of a first frame image; the first frame image is received by the first processor;

[0173] The depth information processing module 1004 is configured to perform depth information processing based on the image data information of the first frame image to obtain depth information of the first frame image.

[0174] The depth information updating module 1006 is configured to update the depth information of the first frame image into metadata of a second frame image based on that the second frame image is received by the first processor and before a frame end interrupt event of the first processor receiving the second frame image is triggered; the metadata of the second frame image is used to be sent to a receiving end by the first processor to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0175] In an embodiment, the image data information obtaining module 1002 is further configured to trigger a frame end interrupt event based on that the first frame image is received by the first processor, and obtain the image data information of the first frame image.

[0176] In an embodiment, the depth information processing module 1004 is further configured to perform depth estimation based on the image data information of the first frame image to obtain the depth information of the first frame image before a frame start interrupt event of the first processor receiving the second frame image is triggered.

[0177] In an embodiment, the depth information updating module 1006 is further configured to update the depth information of the first frame image into the metadata of the second frame image based on that the frame start interrupt event of the first processor receiving the second frame image is triggered and before a frame end interrupt event of the first processor receiving the second frame image is triggered.

[0178] In an embodiment, the first processor comprises an image processor; and the second processor comprises a neural network processor.

[0179] In an embodiment, the image data information obtaining module 1002 is further configured to determine a state of the neural network processor based on that the receiving of the first frame image by the image processor is ended, and obtain the image data information of the first frame image based on that the neural network processor is in a normal working state.

[0180] In an embodiment, the image data information of the first frame image comprises image attribute information and data storage location information of the first frame image; the depth information processing module 1004 is further configured to obtain image data of the first frame image according to the data storage location information, and perform depth estimation based on the image data and the image attribute information of the first frame image to obtain the depth information of the first frame image.

[0181] In one embodiment, the depth information update module 1006 is further configured to determine the metadata of the second frame image based on receiving the second frame image through the first processor; and to add the depth information of the first frame image to the metadata of the second frame image before the first processor finishes receiving the second frame image.

[0182] In one embodiment, the depth information update module 1006 is further configured to determine the depth information field from the metadata of the second frame image; and write the depth information of the first frame image into the depth information field before the first processor finishes receiving the second frame image.

[0183] In one embodiment, such as Figure 11 As shown, an image processing apparatus 1100 is provided, including: a first frame image receiving module 1102, a depth information processing module 1104, and a metadata sending module 1106, wherein:

[0184] The first frame image receiving module 1102 is used to receive the first frame image through the first processor;

[0185] The depth information processing module 1104 obtains the image data information of the first frame image through the second processor, performs depth information processing based on the image data information of the first frame image by the second processor, and updates the metadata of the second frame image with the obtained depth information of the first frame image before the first processor finishes receiving the second frame image.

[0186] The metadata sending module 1106 is used to send the metadata of the second frame image to the receiving end through the first processor, so as to instruct the receiving end to process the first frame image according to the depth information of the first frame image in the metadata of the second frame image and the corresponding first frame image.

[0187] In one embodiment, the system further includes an interval duration determination module, which is used to determine the processing statistics duration of the second processor for depth information processing of the received image; and based on the processing statistics duration, determine the reception interval duration between the first processor receiving two adjacent frames of images; wherein the reception interval duration is greater than or equal to the processing statistics duration.

[0188] In one embodiment, the image further includes a blurring processing module, which is used to determine blurring processing parameters based on the image data and depth information of the first frame image, and to perform image blurring processing on the first frame image using the blurring processing parameters.

[0189] In an embodiment, the image data sending module is further configured to send, based on receiving the first frame image ending by the first processor, image data of the first frame image to the receiving end by the first processor, to instruct the receiving end to process the image data corresponding to the first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0190] In an embodiment, the depth information comprises a depth image of the first frame image, and the attribute information updating module is further configured to obtain attribute information corresponding to the depth image; the first processor is further configured to, based on receiving the second frame image by the first processor and before receiving the second frame image ending by the first processor, update the attribute information into the metadata of the second frame image; and the metadata sending module 1106 is further configured to send, by the first processor, the metadata of the second frame image to the receiving end, to instruct the receiving end to process the image data corresponding to the first frame image according to the depth image of the first frame image and the attribute information in the metadata of the second frame image.

[0191] In an embodiment, as shown in FIG. 12, Figure 12 there is provided an image processing apparatus 1200, comprising: a first frame image receiving module 1202 and a second frame image metadata sending module 1204, wherein:

[0192] The first frame image receiving module 1202 is configured to receive the first frame image and send the first frame image to the receiving end;

[0193] The second frame image metadata sending module 1204 is configured to, based on receiving the second frame image, send the metadata of the second frame image to the receiving end, to instruct the receiving end to process the image data corresponding to the first frame image according to the depth information of the first frame image in the metadata of the second frame image.

[0194] The depth information of the first frame image is obtained by the second processor based on depth information processing of the image data information of the first frame image, and is updated by the second processor into the metadata of the second frame image based on receiving the second frame image and before receiving the second frame image ending.

[0195] The above-mentioned modules in the image processing apparatus can be realized by software, hardware and combinations thereof in whole or in part. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in hardware form, or stored in the memory in the electronic device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned modules.

[0196] In an embodiment, as shown in FIG. 13, Figure 13 there is provided an image processing system 1300, comprising:

[0197] The first processor 1302 is configured to receive a first frame image; based on receiving a second frame image, send metadata of the second frame image to a receiving end, so as to instruct the receiving end to process the corresponding first frame image according to depth information of the first frame image in the metadata of the second frame image;

[0198] The second processor 1304 is configured to obtain image data information of the first frame image; perform depth information processing based on the image data information of the first frame image to obtain depth information of the first frame image; and based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image, update the depth information of the first frame image into the metadata of the second frame image.

[0199] The scheduling processor 1306 is configured to control the first processor to receive the first frame image and the second frame image, and control the first processor to send the metadata of the second frame image to the receiving end.

[0200] The scheduling processor 1306 is further configured to control the second processor to perform depth information processing based on the image data information of the first frame image, and control the second processor to update the depth information of the first frame image into the metadata of the second frame image.

[0201] In the above image processing system, the scheduling processor controls the first processor to receive the first frame image and send the first frame image to the receiving end, and based on receiving the second frame image, sends the metadata of the second frame image to the receiving end, so as to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image. The scheduling processor controls the second processor to obtain image data information of the first frame image received by the first processor, performs depth information processing based on the image data information of the first frame image, and based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image, updates the obtained depth information of the first frame image into the metadata of the second frame image. In the image processing process, the second processor is each time for image data information subjected to depth information processing, which is image data information of an image that has been completely received by the first processor, ensuring consistency of image data information of the same image in different processing processes, thereby improving accuracy of obtained depth information and improving image processing effect when processing based on the depth information. In an embodiment, an electronic device is provided, which can be a server or a terminal, and an internal structure diagram of the electronic device can be as follows: Figure 14As shown in the figure. The electronic device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the electronic device is used to provide computing and control capability. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the electronic device is used to store image processing data. The input / output interface of the electronic device is used to exchange information between the processor and external devices. The communication interface of the electronic device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement an image processing method.

[0202] Those skilled in the art can understand that, Figure 14 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0203] The embodiment of the present application also provides a computer readable storage medium. One or more non-volatile computer readable storage media containing computer executable instructions, when the computer executable instructions are executed by one or more processors, make the processor execute the steps of the image processing method.

[0204] The embodiment of the present application also provides a computer program product containing instructions, when it runs on a computer, makes the computer execute the image processing method.

[0205] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0206] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0207] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0208] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An image processing method, characterized by, The application is applied to a second processor, and comprises: obtaining image data information of a first frame image, wherein the first frame image is obtained by a first processor; performing depth information processing based on the image data information of the first frame image to obtain depth information of the first frame image; updating the depth information of the first frame image into metadata of a second frame image based on the first processor receiving the second frame image and before the first processor finishes receiving the second frame image; wherein the metadata of the second frame image is used for sending to a receiving end by the first processor to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

2. The method of claim 1, wherein, The obtaining of the image data information of the first frame image comprises: triggering a frame end interrupt event based on the first processor receiving the first frame image to obtain the image data information of the first frame image.

3. The method of claim 1, wherein, The performing of the depth information processing based on the image data information of the first frame image to obtain the depth information of the first frame image comprises: performing depth estimation based on the image data information of the first frame image to obtain the depth information of the first frame image before the first processor triggers a frame start interrupt event of receiving the second frame image.

4. The method of claim 1, wherein, The updating of the depth information of the first frame image into the metadata of the second frame image based on the first processor receiving the second frame image and before the first processor finishes receiving the second frame image comprises: updating the depth information of the first frame image into the metadata of the second frame image based on the first processor triggering the frame start interrupt event of receiving the second frame image and before the first processor triggering a frame end interrupt event of receiving the second frame image.

5. The method according to any one of claims 1 to 4, characterized in that, The first processor comprises an image processor, and the second processor comprises a neural network processor.

6. The method of claim 5, wherein, The obtaining of the image data information of the first frame image comprises: determining a state of the neural network processor based on the first processor receiving the first frame image; obtaining the image data information of the first frame image based on the neural network processor being in a normal working state.

7. The method according to any one of claims 1 to 4, characterized in that, The image data information of the first frame image comprises image attribute information and data storage location information of the first frame image. The performing of the depth information processing based on the image data information of the first frame image to obtain the depth information of the first frame image comprises: obtaining image data of the first frame image according to the data storage location information; performing depth estimation according to the image data of the first frame image and the image attribute information to obtain the depth information of the first frame image.

8. The method according to any one of claims 1 to 4, characterized in that, The updating of the depth information of the first frame image into the metadata of the second frame image based on the first processor receiving the second frame image and before the first processor finishes receiving the second frame image comprises: determining the metadata of the second frame image based on the first processor receiving the second frame image. adding the depth information of the first frame image into the metadata of the second frame image before the first processor finishes receiving the second frame image.

9. The method of claim 8, wherein, The adding the depth information of the first frame image into the metadata of the second frame image before the first processor finishes receiving the second frame image comprises: determining a depth information field from the metadata of the second frame image; writing the depth information of the first frame image into the depth information field before the first processor finishes receiving the second frame image.

10. An image processing method characterized by, Applied to a scheduling processor, comprising: receiving a first frame image by a first processor; obtaining image data information of the first frame image by a second processor, so that the second processor performs depth information processing based on the image data information of the first frame image, and updates the obtained depth information of the first frame image into the metadata of a second frame image before the first processor finishes receiving the second frame image based on receiving the second frame image by the first processor; sending the metadata of the second frame image to a receiving end by the first processor, so as to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image and the corresponding first frame image.

11. The method of claim 10, wherein, The method further comprises: determining a processing statistical duration of the second processor for depth information processing of the received image; based on the processing statistical duration, determining a receiving interval duration between adjacent two frame images received by the first processor; the receiving interval duration is greater than or equal to the processing statistical duration.

12. The method of claim 10, wherein, The method further comprises: based on the image data of the first frame image and the depth information, determining a virtualization processing parameter; performing image virtualization processing on the first frame image by the virtualization processing parameter.

13. The method of claim 10, wherein, The method further comprises: based on the end of receiving the first frame image by the first processor, sending the image data of the first frame image to the receiving end by the first processor, so as to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image and the image data of the corresponding first frame image.

14. The method according to any one of claims 10 to 13, characterized in that, The depth information comprises a depth image of the first frame image; the method further comprises: obtaining attribute information corresponding to the depth image; based on receiving the second frame image by the first processor and before the first processor finishes receiving the second frame image, updating the attribute information into the metadata of the second frame image by the first processor; The sending the metadata of the second frame image to the receiving end by the first processor, so as to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image and the corresponding first frame image, comprises: sending the metadata of the second frame image to the receiving end by the first processor, so as to instruct the receiving end to process the corresponding first frame image according to the depth image and attribute information of the first frame image in the metadata of the second frame image and the corresponding first frame image.

15. An image processing method, characterized by, Applied to a first processor, comprising: receive a first frame image and send the first frame image to a receiving end; based on receiving a second frame image, send metadata of the second frame image to the receiving end to instruct the receiving end to process the corresponding first frame image according to depth information of the first frame image in the metadata of the second frame image; wherein the depth information of the first frame image is obtained by a second processor based on image data information of the first frame image, and the second processor updates the obtained depth information of the first frame image into the metadata of the second frame image based on receiving the second frame image and before the end of receiving the second frame image.

16. An image processing apparatus characterized by comprising: comprising: an image data information acquisition module for acquiring image data information of a first frame image; the first frame image is obtained by a first processor; a depth information processing module for processing depth information based on image data information of the first frame image to obtain depth information of the first frame image; a depth information updating module for updating the depth information of the first frame image into the metadata of the second frame image based on receiving a second frame image by the first processor and before the end of receiving the second frame image by the first processor; wherein the metadata of the second frame image is used to send to the receiving end by the first processor to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

17. An image processing apparatus characterized by comprising: comprising: a first frame image receiving module for receiving a first frame image by a first processor; a depth information processing module for obtaining image data information of the first frame image by a second processor, processing depth information based on the image data information of the first frame image by the second processor, and updating the obtained depth information of the first frame image into the metadata of the second frame image based on receiving a second frame image by the first processor and before the end of receiving the second frame image by the first processor; a metadata sending module for sending the metadata of the second frame image to the receiving end by the first processor to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image.

18. An image processing apparatus characterized by comprising: comprising: a first frame image receiving module for receiving a first frame image and sending the first frame image to a receiving end; a second frame image metadata sending module for sending metadata of a second frame image to the receiving end based on receiving the second frame image to instruct the receiving end to process the corresponding first frame image according to depth information of the first frame image in the metadata of the second frame image; The depth information of the first frame image is obtained by the second processor based on image data information of the first frame image, and the second processor updates the obtained depth information of the first frame image into metadata of the second frame image based on receiving the second frame image and before the end of receiving the second frame image.

19. An image processing system, characterized by Comprise: A first processor configured to receive a first frame image; The metadata of the second frame image is sent to a receiving end based on receiving the second frame image, so as to instruct the receiving end to process the corresponding first frame image according to the depth information of the first frame image in the metadata of the second frame image; A second processor configured to obtain image data information of the first frame image; The depth information of the first frame image is obtained by the second processor based on image data information of the first frame image, and the second processor updates the obtained depth information of the first frame image into metadata of the second frame image based on receiving the second frame image and before the end of receiving the second frame image. A scheduling processor configured to control the first processor to receive the first frame image and the second frame image, and control the first processor to send the metadata of the second frame image to the receiving end; The scheduling processor is also configured to control the second processor to process the depth information based on the image data information of the first frame image, and control the second processor to update the depth information of the first frame image into the metadata of the second frame image. The computer program is executed by the processor, so that the processor executes the steps of the image processing method in any one of claims 1 to 15.

20. An electronic device comprising a memory and a processor, said memory having stored therein a computer program, characterised in that, The computer program is executed by the processor, so that the processor executes the steps of the method in any one of claims 1 to 15.

21. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor, so that the processor executes the steps of the method in any one of claims 1 to 15.

22. A computer program product comprising a computer program, characterized in that, ​

Images