Image processing method, image processing apparatus, and image processing system

Abstract

The invention provides an image processing method, an image processing device and an image processing system, and relates to the technical field of image processing. The image processing method comprises the following steps: inputting an image into a viewpoint generation model; the image is pre-processed and depth estimated based on a viewpoint generation model to generate a multi-viewpoint image of the image. In the embodiment of the invention, pretreatment, depth estimation and viewpoint image generation are integrated into the same model, and can be configured and loaded in the same processor, such as an image processor, and the image processor can directly calculate and generate the multi-viewpoint image on the input original image based on the viewpoint generation model, so that the multi-viewpoint image generation efficiency is improved. According to the scheme, the data transmission between the image processor and other processors (such as a main processor) can be reduced, the parallel reasoning capability of the model can be improved, the hardware resource consumption can be reduced, and the viewpoint graph generation speed and efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image processing method, an image processing apparatus, and an image processing system. Background Art

[0002] Three-dimensional (3D) multi-viewpoint images are a technology for capturing or generating three-dimensional scenes through multiple viewpoints, which can provide a more realistic and immersive visual experience. Different from traditional two-dimensional images, 3D multi-viewpoint images not only contain color and texture information, but also depth information, enabling users to observe the scene from different angles and feel the three-dimensional spatial relationship. This technology has extensive applications in fields such as Virtual Reality (VR), Augmented Reality (AR), free viewpoint video, and 3D modeling.

[0003] The generation and calculation of 3D multi-viewpoint images require a large amount of computing resources. How to improve the generation efficiency of multi-viewpoint images and reduce resource consumption is an urgent technical problem in this field. Summary of the Invention

[0004] This application provides an image processing method, an image processing apparatus, and an image processing system, which can improve the generation efficiency of multi-viewpoint images and reduce resource consumption.

[0005] In a first aspect, an image processing method is provided, including: inputting an image into a viewpoint generation model, and performing preprocessing and depth estimation on the image based on the viewpoint generation model to generate a multi-viewpoint image of the image.

[0006] In the embodiments of this application, preprocessing, depth estimation, and viewpoint image generation are integrated into the same model, which can be configured to be loaded in the same processor, such as an image processor. The image processor can directly calculate and generate a multi-viewpoint image of the input original image based on the viewpoint generation model. This solution can not only reduce data transmission between the image processor and other processors (such as the main processor), which is beneficial to improving the parallel inference ability of the model, but also reduce hardware resource consumption and improve the speed and efficiency of viewpoint map generation.

[0007] In some possible embodiments, the viewpoint generation model includes a first viewpoint generation sub-model and a second viewpoint generation sub-model. Among them, the pre-processing and depth estimation of the image based on the viewpoint generation model to generate the multi-viewpoint image of the image includes: respectively performing pre-processing and depth estimation on the image based on the first viewpoint generation sub-model and the second viewpoint generation sub-model to parallelly generate the first viewpoint image and the second viewpoint image of the image; wherein, the first viewpoint image and the second viewpoint image are respectively the left viewpoint image and the right viewpoint image, or, the first viewpoint image and the second viewpoint image are respectively the upper viewpoint image and the lower viewpoint image.

[0008] In the embodiments of the present application, the two sub-models in the viewpoint generation model can parallelly process the original image to generate two types of viewpoint images. Compared with the process of sequentially generating multiple viewpoint images from the original image in series, the generation effect and generation speed of the viewpoint images can be effectively improved.

[0009] In some possible embodiments, the method further includes: inputting the first viewpoint image into the first viewpoint generation sub-model, and processing the first viewpoint image based on the first viewpoint generation sub-model to generate the third viewpoint image of the image; using the second viewpoint image to input into the second viewpoint sub-model, and processing the second viewpoint image based on the second viewpoint generation sub-model to generate the fourth viewpoint image of the image.

[0010] Through the technical solution of this embodiment, in the process of generating multiple viewpoint images, the previous viewpoint image is used as the input for generating the next viewpoint image. These multiple viewpoint images can be of the same type of viewpoint images. For example, they all belong to the left viewpoint image or the right viewpoint image, etc., which is beneficial to further improving the generation effect and speed of the viewpoint images.

[0011] In some possible embodiments, the above-mentioned inputting the image into the viewpoint generation model includes: merging multiple images, and inputting the merged multiple images into the viewpoint generation model. The above-mentioned pre-processing and depth estimation of the image based on the viewpoint generation model to generate the multi-viewpoint image of the image includes: based on the viewpoint generation model, parallelly performing pre-processing and depth estimation on the merged multiple images to form the multi-viewpoint image of the multiple images.

[0012] In some possible embodiments, the above-mentioned pre-processing and depth estimation of the image based on the viewpoint generation model to generate the multi-viewpoint image of the image includes: loading the viewpoint generation model multiple times to parallelly perform pre-processing and depth estimation on multiple images to form the multi-viewpoint image of the multiple images.

[0013] Through the technical solutions of the above two embodiments, the generation of viewpoint images for multiple images can be performed in parallel, thereby further improving the generation speed and efficiency of the viewpoint images.

[0014] In some possible embodiments, the method further includes: storing the multi-viewpoint image in a shared memory, and feeding back the storage identifier of the multi-viewpoint image in the shared memory to the main processor, so that the main processor obtains the multi-viewpoint image according to the storage identifier and feeds it back to the service system in the service side.

[0015] Through the technical solution of the embodiments of the present application, based on storing the multi-viewpoint image in the shared memory, the main processor can feed back the multi-viewpoint image to the service system by transmitting the storage identifier of the multi-viewpoint image in the shared memory, without storing the multi-viewpoint image in the main processor, which can save the storage space of the main processor and is also beneficial to reducing the data transmission between the main processor and the image processor, saving the consumption of hardware resources.

[0016] In some possible embodiments, the method further includes: copying the multi-viewpoint image to the locked-page memory of the main processor in the service side, so that the main processor in the service side feeds back the multi-viewpoint image in the locked-page memory to the service system in the service side.

[0017] In the technical solution of the embodiments of the present application, the locked-page memory is an efficient system memory, which can realize the fast storage and reading of the viewport image therein, and can effectively improve the speed at which the user obtains the viewport image.

[0018] In some possible embodiments, the method further includes: during the process of generating each viewport image in the multi-viewpoint image, synchronously copying each viewport image to the shared memory or the locked-page memory of the main processor in the service side.

[0019] Through the technical solution of this embodiment, while generating the viewport image, synchronously copying the viewport image is beneficial to further shortening the time for the user to obtain the viewport image.

[0020] In some possible embodiments, the method is applied to an image processor, and the method further includes: multiple image processors sending the historical image processing tasks to be processed and / or the running states to the main processor, so that the main processor allocates new image processing tasks according to the historical image processing tasks to be processed in the multiple image processors and / or the running states of the multiple image processors.

[0021] Through the technical solution of the embodiments of the present application, the processes can be managed in real time according to the current number of tasks (service requests), and a single process can execute multiple tasks in parallel, so as to optimize the resource utilization in the image processor and improve the speed of the image processor for model inference and generating viewport images.

[0022] In some possible embodiments, the method is applied to an image processor, and the method further includes: multiple image processors sending the historical image processing tasks to be processed and / or the running states to the main processor, so that the main processor allocates new image processing tasks according to the historical image processing tasks to be processed in the multiple image processors and / or the running states of the multiple image processors.

[0023] The technical solution of the embodiment of the present application can achieve resource balance of multiple image processors, ensure that each image processor has a high utilization rate, and thus is conducive to improving the overall performance of the image processing device.

[0024] In some possible implementation manners, the main processor has an image processing task queue, and the main processor is configured to rearrange multiple image processing tasks in the image processing task queue according to the priorities of newly received image processing tasks.

[0025] In some possible implementation manners, the method further includes: multiple image processors generate an image output task queue of multi-viewpoint images, and feedback each image output task in the image output task queue to the main processor. The main processor is configured to compare the identifiers of the image output tasks with the identifiers of the input image processing tasks, and output the multi-viewpoint images carried in the image output tasks to the service system.

[0026] In some possible implementation manners, the method further includes: each of the multiple image processors outputs the generated multi-viewpoint images to the service system.

[0027] In the technical solutions of the above two implementation manners, the synchronous callback of the multi-viewpoint images to the service system is conducive to the orderly execution and output of multiple image processing tasks in the system, and is convenient for uniformly managing multiple image processing tasks. The asynchronous callback of the multi-viewpoint images to the service system is conducive to accelerating the speed of feedback of the multi-viewpoint images to the service system and the user.

[0028] In some possible implementation manners, the main processor is configured to obtain the number of multiple image processors for performing multi-viewpoint image generation, and the number of processes of each of the multiple image processors; the main processor is configured to map the process identifiers and image processor identifiers in each image processor, and configure the environment variables in each image processor according to the image processor identifiers and load the viewpoint generation model to generate multi-viewpoint images.

[0029] In this technical solution, the main processor can automatically detect the number of image processors available for generating viewpoint images, manage and configure the corresponding image processors, so that the image processors can automatically create viewpoint map generation tasks without modifying and adapting the task codes.

[0030] In some possible embodiments, the above preprocessing and depth estimation of an image based on a viewpoint generation model to generate a multi-viewpoint image of the image include: performing a first preprocessing and a third preprocessing on the image based on the viewpoint generation model; performing depth estimation on the image after the first preprocessing based on the viewpoint generation model, and performing a second preprocessing on the image after the depth estimation; performing viewpoint generation based on the viewpoint generation model, the image after the second preprocessing, and the image after the third preprocessing to generate a multi-viewpoint image; wherein at least one of the first preprocessing, the second preprocessing, and the third preprocessing includes at least one of the following processes: size scaling, normalization, or standardization.

[0031] In a second aspect, there is provided an image processing apparatus, including: an image processor configured to execute the image processing method according to the first aspect or any possible embodiment of the first aspect.

[0032] In some possible embodiments, the image processing apparatus further includes: a main processor connected to at least one image processor and configured to control at least one image processor.

[0033] In a third aspect, there is provided an image processing system, including: an algorithm side and a service side, the algorithm side including the image processing apparatus according to the second aspect or any possible embodiment of the second aspect; the service side is configured to receive an image input by a service system and transmit the image to the algorithm side to generate a multi-viewpoint image of the image, and the algorithm side is configured to feed back the generated multi-viewpoint image to the service system in the service side. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 FIG. shows a schematic diagram of an image processing system provided by an embodiment of the present application.

[0035] Figure 2 FIG. shows a schematic diagram of another image processing system provided by an embodiment of the present application.

[0036] Figure 3 FIG. shows a schematic diagram of an image processing method provided by an embodiment of the present application.

[0037] Figure 4 FIG. shows a schematic diagram of a viewpoint generation model provided by an embodiment of the present application.

[0038] Figure 5 FIG. shows a schematic diagram of another viewpoint generation model provided by an embodiment of the present application.

[0039] Figure 6 FIG. shows a schematic diagram of the generation sequence of a 9-viewpoint image provided by an embodiment of the present application.

[0040] Figure 7Shows a schematic diagram of an original image, a depth image, and a generated 9-viewpoint image provided by an embodiment of the present application.

[0041] Figure 8 Shows a schematic diagram of a viewpoint image generation provided by an embodiment of the present application.

[0042] Figure 9 Shows a schematic diagram of another viewpoint image generation provided by an embodiment of the present application.

[0043] Figure 10 Shows another schematic diagram of an image processing device provided by an embodiment of the present application.

[0044] Figure 11 Shows a schematic diagram of a shared memory in a GPU provided by an embodiment of the present application.

[0045] Figure 12 Shows another schematic diagram of an image processing device provided by an embodiment of the present application.

[0046] Figure 13 Shows a schematic diagram of a GPU provided by an embodiment of the present application.

[0047] Figure 14 Shows another schematic diagram of an image processing device provided by an embodiment of the present application.

[0048] Figure 15 Shows another schematic diagram of an image processing device provided by an embodiment of the present application.

[0049] Figure 16 Shows a schematic diagram of a resource scaling management provided by an embodiment of the present application.

[0050] Figure 17 Shows a schematic diagram of an image processing system provided by an embodiment of the present application. Detailed implementation manners

[0051] Next, the technical solutions in the present application will be described in conjunction with the accompanying drawings.

[0052] The present application relates to a technical solution for generating 3D multi-viewpoint images based on an algorithm model. First, in combination with Figures 1 to 2 A brief introduction to the possible application scenarios of the embodiments of the present application will be given.

[0053] Figure 1An image processing system provided by an embodiment of the present application is shown. The image processing system includes a service - end device and an algorithm - end device. Among them, the service - end device may include various intelligent terminals such as mobile phones, personal computers, AR / VR devices, smart wearable devices, in - vehicle devices, etc. The service - end device can be the initiator of image - processing requirements. For example, it can send requests such as 3D multi - viewpoint image generation.

[0054] The algorithm - end device can be configured with relevant algorithm models for performing image processing. For example, machine - learning models, deep - learning models, etc. The algorithm - end device may include devices or servers with data - processing functions such as cloud servers, network servers, application servers, and management servers. The algorithm - end device receives data such as images from the service - end device through an interaction interface, and then performs image processing through machine learning, deep learning, etc. by means of a memory for storing data and a processor for image processing.

[0055] In Figure 1 the shown image processing system, the service - end device can obtain an original image and send the original image to the algorithm - end device, so that the algorithm - end device performs model inference based on the algorithm model, thereby processing the original image to generate a corresponding 3D multi - viewpoint image. The multi - viewpoint image can be fed back to the service - end device through the interaction interface and then to the service system in the service - end device. The user can obtain the multi - viewpoint image through the service system.

[0056] Figure 2 Another image processing system provided by an embodiment of the present application is shown. In Figure 2 this system, the service - end device and the algorithm - end device can be integrated into an integrated device. The user can initiate an image - processing request in this device. The device can directly perform image processing on the original image, generate a corresponding 3D multi - viewpoint image, and feed it back to the user.

[0057] Figure 1 and Figure 2 The processor in this system can support model inference for running a neural - network model or other models (for example, a model based on a support vector machine), and process and calculate the original image. The processor may include, for example, one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), or Neural Network Processing Unit.

[0058] Figure 3The figure shows a schematic diagram of an image processing method provided by an embodiment of the present application. Optionally, the image processing method can be executed on an image processing device. Exemplarily, the image processing device can include an image processor, and the image processing method can be executed on the image processor.

[0059] As Figure 3 shown, the image processing method 10 can include the following steps.

[0060] S11: Input the image into the view point generation model.

[0061] S12: Perform pre-processing and depth estimation on the image based on the view point generation model to generate multi-viewpoint images of the image.

[0062] In the embodiments of the present application, the image processor for executing the above image processing method includes but is not limited to a GPU, and can also be other processors such as a TPU that can be used to process image data. For the convenience of description, the following embodiments take the GPU as an example, and the solutions of other types of image processors can refer to the relevant descriptions of the following embodiments.

[0063] The view point generation model can be configured in the image processor. The view point generation model can include machine learning models, deep learning models, computer vision models, etc. that are suitable for image processing. In addition, the view point generation model can include multiple modules such as pre-processing, depth estimation, and view point image generation.

[0064] As an example, Figure 4 the figure shows a schematic diagram of a view point generation model provided by an embodiment of the present application.

[0065] As Figure 4 shown, the view point generation model can perform pre-processing 1, depth estimation, pre-processing 2, pre-processing 3, and view point generation on the input image (also called the original image) to form multi-viewpoint images corresponding to the original image.

[0066] The input of pre-processing 1 is the original image that needs to perform multi-viewpoint generation. The original image can be a color image (also called an RGB image). Pre-processing 1 mainly realizes performing at least one of the following processes on the image:

[0067] (1) Perform size scaling (to meet the input size required by the depth estimation model);

[0068] (2) Normalize the image (for example, the original image pixel data can be divided by 255 to convert the image into data between 0 and 1);

[0069] (3) Perform standardization (that is, subtract the mean and divide by the variance).

[0070] After processing the original image in the above three steps in sequence, a standardized image can be output as the input of the depth estimation model.

[0071] The depth estimation module mainly performs depth prediction on the image. The output of preprocessing 1 is used as the input for the calculation of the depth estimation module. After the depth estimation module completes the calculation of the input image data, a depth estimation image of the original image is obtained.

[0072] Preprocessing 2 is to scale the depth estimation image and then perform normalization again so that the output image meets the input size of the subsequent view point generation module.

[0073] Preprocessing 3 is to scale and normalize the original image, and the output image also meets the input size of the view point generation module.

[0074] The images after preprocessing 2 and preprocessing 3 can be stitched according to the channel dimension to generate the input for the view point generation module.

[0075] The view point generation module can generate a view point image based on the depth image after preprocessing 2 and the original image after preprocessing 3.

[0076] In some embodiments, the above view point generation module and depth estimation module may include algorithm models.

[0077] In the processing of some image data, the preprocessing process (including the above preprocessing 1, 2, and 3) is executed in the CPU, while the depth estimation and view point image generation processes are executed in the GPU. The data will be frequently switched between the CPU and the GPU, which will not only cause waste of resources but also reduce the image processing speed. In addition, in this embodiment, the models in the GPU and the preprocessing in the CPU generally use different software frameworks. For example, the model in the GPU is implemented through pytorch, and the image preprocessing in the CPU is implemented through the image processing library opencv. Therefore, there are certain obstacles to the transmission or copying of data between the two frameworks. On the other hand, dividing the view point generation process into two parts for processing increases the complexity of the algorithm inference process and cannot fully utilize the parallel inference ability of the model.

[0078] In the embodiment of the present application, the preprocessing, depth estimation, and view point image generation are integrated into the same model, which is configured to be loaded in the GPU. The GPU can directly calculate and generate multi-view point images based on the view point generation model for the input original image. This solution can not only reduce the data transmission between the CPU and the GPU, is beneficial to improving the parallel inference ability of the model, but also can reduce the consumption of hardware resources and improve the speed and efficiency of view point map generation.

[0079] Optionally, in some embodiments, pytoch algorithms (such as F.interpolate) can be used for pre-processing and other operations. The overall algorithm process can be implemented using pytoch, and then the model can be exported as a whole to facilitate model deployment and management. To further improve the performance of the model, the exported model can be optimized as a whole using the tensorrt deployment tool, so that the model has multi-batch and multi-instance high-inference performance.

[0080] Figure 5 A schematic diagram of another viewpoint generation model provided by an embodiment of the present application is shown.

[0081] like Figure 5 As shown, the viewpoint generation model may include a first viewpoint generation sub-model and a second viewpoint generation sub-model, and based on the first viewpoint generation sub-model and the second viewpoint generation sub-model, the image is pre-processed and the depth is estimated respectively to generate the first viewpoint image and the second viewpoint image of the image in parallel. Optionally, the first viewpoint image and the second viewpoint image may be different types of viewpoint images, for example, the first viewpoint image and the second viewpoint image may be a left viewpoint image and a right viewpoint image, or the first viewpoint image and the second viewpoint image may be an upper viewpoint image and a lower viewpoint image.

[0082] In a three-dimensional space, the 9 viewpoint images may be arranged in a 3×3 grid. The 9 viewpoint images may include upper left, upper middle, upper right, middle left, center, middle right, lower left, lower middle, and lower right. Among the 9 viewpoint images, the center viewpoint image may be the original image, and the other 8 viewpoint images may be divided into two groups, which are generated by the first viewpoint generation sub-model and the second viewpoint generation sub-model respectively.

[0083] In an embodiment of the present application, the two sub-models in the viewpoint generation model can process the original image in parallel to generate two types of viewpoint images. Compared with the process of serially generating multiple viewpoint images from the original image, the generation effect and generation speed of the viewpoint image can be effectively improved.

[0084] In the above two viewpoint generation sub-models, each sub-model includes the process of pre-processing, depth estimation and viewpoint generation. For example, the pre-processing in each sub-model can include Figure 4 Pre-processing 1, 2, 3 shown. However, in different sub-models, at least one step of pre-processing, depth estimation and viewpoint generation may be different, for example, at least one pre-processing process is different, so that viewpoint images of different viewing angles can be generated in a targeted manner. The different sub-models can share processing modules with the same processing method, for example, the depth estimation module can be applicable to different sub-models.

[0085] Figure 6 A schematic diagram of a generation sequence of 9-viewpoint images provided in an embodiment of the present application is shown.

[0086] Combined with Figure 5 and Figure 6 As shown, the original image can generate two sets of viewpoint images in parallel through the above two sub-models, and each set of viewpoint images can be the same type of viewpoint images. As an example, Figure 6 the first set of viewpoint images shown in includes viewpoints 1 to 4, and the second set of viewpoint images includes viewpoints 6 to 9. During the generation of the two sets of viewpoint images, the previous viewpoint image can be used as the input of the sub-model to generate the next viewpoint image. For example, during the generation of the first set of viewpoint images, viewpoint 1 can be used as the input of the first viewpoint generation sub-model to generate viewpoint 2, and viewpoint 2 is further used as the input of the first viewpoint generation sub-model to generate viewpoint 3, and so on.

[0087] Optionally, in the embodiments of the present application, viewpoints 1 to 4 can be referred to as left viewpoint images, and viewpoints 6 to 9 can be referred to as right viewpoint images. Viewpoint 5 is the original image.

[0088] As an example, Figure 7 shows a schematic diagram of an original image, a depth image, and the generated 9-viewpoint images provided by the embodiments of the present application.

[0089] Through the technical solution of this embodiment, during the generation of multiple viewpoint images, the previous viewpoint image is used as the input for generating the next viewpoint image. These multiple viewpoint images can be of the same type, for example, all belonging to left viewpoint images or right viewpoint images, etc., which is beneficial to further improving the generation effect and speed of the viewpoint images.

[0090] In some embodiments, the image processing method includes: merging multiple images, and inputting the merged multiple images into a viewpoint generation model; based on the viewpoint generation model, performing pre-processing and depth estimation on the merged multiple images in parallel to form multi-viewpoint images of the multiple images.

[0091] Optionally, in the case where the viewpoint generation model includes the above-mentioned first viewpoint generation sub-model and the second viewpoint generation sub-model, Figure 8 shows a schematic diagram of viewpoint image generation provided by the embodiments of the present application. As shown in Figure 8 each sub-model can also be used to receive multiple images, merge the multiple images, and perform batch processing on the merged multiple images to obtain the same type of viewpoint images of each image in the multiple images.

[0092] In some embodiments, the image processing method includes: loading the viewpoint generation model multiple times, so as to perform pre-processing and depth estimation on multiple images in parallel to form multi-viewpoint images of the multiple images.

[0093] Optionally, in the case where the viewpoint generation model includes the above-mentioned first viewpoint generation sub-model and second viewpoint generation sub-model, Figure 9 FIG. shows a schematic diagram of generating another viewpoint image provided by an embodiment of the present application, as Figure 9 shown, the two viewpoint generation sub-models loaded multiple times can form multiple processes running in parallel in the GPU, and each process can be used to perform image processing on one image to generate the same type of viewpoint image of the image.

[0094] In some other embodiments, the GPU can adopt the above two methods simultaneously, that is, the model is loaded multiple times, and among the multiple models loaded multiple times, at least one model can perform combined batch processing on multiple images.

[0095] Through the technical solutions of the above embodiments, it is possible to generate viewpoint images for multiple images in parallel, thereby further improving the generation speed and efficiency of the viewpoint map.

[0096] In the above embodiments, the case where the viewpoint generation model includes two viewpoint generation sub-models is taken as an example for illustration. Optionally, the viewpoint generation model can also include three or more sub-models, and each sub-model can be used to generate a type of viewpoint image. For example, it can include left viewpoint images, middle viewpoint images, and right viewpoint images, or it can also include upper viewpoint images, middle viewpoint images, and lower viewpoint images.

[0097] Figure 10 FIG. shows a schematic diagram of an image processing apparatus provided by an embodiment of the present application.

[0098] As Figure 10 shown, the image processing apparatus 100 may include an image processor 110 and a main processor 120. In an embodiment of the present application, the image processing apparatus 100 may simultaneously include the service side and the algorithm side of the image processing system. In other words, in this image processing system, the service side and the algorithm side can share the image processor 110 and the main processor 120.

[0099] Among them, the image processor 110 includes a shared memory 111, and the image processing method may further include: storing a multi-viewpoint image through the shared memory 111, and the storage identifier of the multi-viewpoint image in the shared memory 111 is used to feedback to the main processor 120, so that the main processor 120 obtains the multi-viewpoint image according to the storage identifier and feedbacks it to the service system in the service side.

[0100] Optionally, the main processor 120 includes, but is not limited to, a processing device with control and computing functions such as a CPU, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), etc. For the convenience of description, the CPU is taken as an example in the following embodiments, and the solutions for other types of main processors can refer to the relevant descriptions of the following embodiments.

[0101] In the embodiments of the present application, after an image processor, such as a GPU, generates a multi-viewpoint image based on an original image, the multi-viewpoint image can be stored in its shared memory. In addition to being accessible by the GPU, the main processor, such as a CPU, can also access this shared memory. In some embodiments, the GPU can send the storage identifier of the multi-viewpoint image in its shared memory to the CPU, so that the CPU can obtain the multi-viewpoint image in the shared memory of the GPU according to this storage identifier. As an example, the storage identifier of the multi-viewpoint image in the shared memory may include an Http handle.

[0102] Figure 11 A schematic diagram of a shared memory in a GPU provided by the embodiments of the present application is shown.

[0103] As Figure 11 shown, the service side performs hardware decoding on the image to be processed. After the decoded image is stored in the shared memory (or also called video memory) of the GPU, the GPU video memory handle can be transmitted through the Http protocol. The algorithm side obtains data through the handle for model inference calculation, and then directly puts the calculated multi-viewpoint image into the already allocated shared memory result of the GPU. The video memory handle of this shared memory result can be fed back to the caller. Optionally, the above-mentioned GPU shared memory can be a Cuda-based shared memory.

[0104] In this solution, since the input or output of the viewpoint generation algorithm is often an image with high resolution, especially when the output is a synthesis of 9 viewpoint images, data transmission performance is an issue that has to be considered. When the service side and the algorithm side are deployed on one server, to avoid unnecessary performance loss caused by data copying, the solution of using the GPU shared memory in this scheme can effectively avoid the problem of data copy transmission, and effectively improve the overall performance of the algorithm service and the business service when they are on one server.

[0105] Figure 11When the service end receives an image or video, it can use the GPU for image or video decoding, that is, the hardware decoding module of the image. Therefore, the decoded data is in the GPU memory. In this solution, the service end can allocate a GPU shared memory and directly put the decoded data into the shared memory, and exchange data by transmitting the handle of the shared memory through http. The algorithm service calculates the viewpoint generation by parsing the request parameters to obtain the GPU shared memory and then obtaining the data. At the same time, an output shared memory is allocated, and the shared memory handle of the output result is fed back to the service calling end.

[0106] Through the technical solution of the embodiment of the present application, a shared memory is set in the GPU to store multi-viewpoint images. The CPU can be fed back multi-viewpoint images by transmitting the storage identifier of the multi-viewpoint images in the shared memory, without storing the multi-viewpoint images in the CPU, which can save the storage space of the CPU and is also beneficial to reducing the data transmission between the CPU and the GPU and saving the consumption of hardware resources.

[0107] Figure 12 Another schematic diagram of the image processing device provided by the embodiment of the present application is shown.

[0108] As Figure 12 shown, the image processing device 100 is used as the algorithm end, in which an image processor 110 is provided. Another image processing device 200 is used as the service end, in which a main processor 210 is provided. In the embodiment of the present application, the service end and the algorithm end in the image processing system are separately set and can be two independent electronic devices.

[0109] In the image processing device 200 (i.e., the service end), the main processor 210 is, for example, a CPU, and a pinned memory 211 can be set in the main processor 210. After the GPU in the image processing device 100 (i.e., the algorithm end) generates multi-viewpoint images, in addition to being stored in the memory 112 of the GPU, the multi-viewpoint images can also be copied and stored in the pinned memory 211 of the main processor 210 of the image processing device 200. The main processor 210 can feed back the multi-viewpoint images in the pinned memory 211 to the service system. That is, in this embodiment, the image processing method executed by the image processor 110 may include: copying the multi-viewpoint images to the pinned memory of the main processor in the service end, so that the main processor in the service end feeds back the multi-viewpoint images in the pinned memory to the service system in the service end.

[0110] In this solution, when the service side and the algorithm side are not on the same GPU node. First, two pages of pinned memory can be allocated in the CPU of the service side, and the received image data is directly placed into the pinned memory, and then the image is copied to the GPU memory, and the GPU performs model inference calculation. On the other hand, the caller needs to feedback data, and the multi-viewpoint images obtained by GPU calculation can also be synchronously copied to the pinned memory of the CPU.

[0111] In the technical solution of the embodiment of the present application, the pinned memory is a kind of efficient system memory, which can realize the fast storage and reading of the viewpoint images therein, and can effectively improve the speed at which users obtain the viewpoint images.

[0112] For the above Figure 10 and Figure 12 In the technical solution of the embodiment shown, in some embodiments, during the process of the image processor 110 generating the multi-viewpoint image, the multi-viewpoint image can be synchronously copied to the shared memory 111 of the image processor 110 or the pinned memory 211 of the main processor 210 in the service side.

[0113] For example, referring back to Figure 6 In the embodiment shown, after the GPU parallel generates viewpoints 1 and 6, viewpoints 1 and 6 can be copied to the shared memory or the pinned memory. During the copying process, the GPU can synchronously continue to generate viewpoint 2 based on viewpoint 1, and at the same time, it can also synchronously generate viewpoint 7 based on viewpoint 6. And so on, the GPU can complete the copying of the 9-viewpoint image, and the storage method of the 9-viewpoint image in the shared memory or the pinned memory can be seen in Figure 6 the right part shown in.

[0114] Through the technical solution of this embodiment, while generating the viewpoint images, synchronously copying the viewpoint images is beneficial to further shorten the time for users to obtain the viewpoint images.

[0115] In order to further improve the execution efficiency of the viewpoint generation model algorithm in the above image processor (such as GPU), the embodiment of the present application can implement parallel processing and dynamic batching solutions of multiple graphics cards, multiple processes, and multiple instances in a single server. Figure 13 Shows a schematic diagram of a GPU provided by an embodiment of the present application.

[0116] As Figure 13 shown, the GPU can load the viewpoint generation model at least once to start at least one process, and the GPU can obtain the image processing tasks to be executed in real time and manage the started processes according to the image processing tasks to be executed.

[0117] In the embodiments of the present application, a single GPU (or a single graphics card) supports batch parallel inference for multiple images. As an example, a single GPU can support batch processing of 16 images simultaneously. The quantity of this batch processing is related to the hardware design of the GPU, and the embodiments of the present application do not make specific limitations on it.

[0118] When the GPU loads the viewpoint generation model once, a process can be started. As Figure 13 shown, when the GPU loads the viewpoint generation model multiple times, multiple processes can be started in parallel. Each process can execute the processing of one or more images in parallel, that is, the generation of viewpoint images.

[0119] In addition, the GPU can also manage image processing tasks (also called service requests). The GPU can obtain the image processing tasks to be processed in real time, form an image processing task queue (also called a service request queue), and start one or more processes according to one or more image processing tasks in the image processing task queue, so as to execute the tasks (such as resource download, data acquisition, pre-processing of model inference, model inference, and post-processing of model inference, etc.). The data of the service request can be put into the model request queue. The model inference process can monitor the queue periodically (the cycle time can be the maximum waiting time for model inference). For batch inference of processes, in one execution cycle, the data in the queue will form a batch process for parallel inference, improving the throughput of model inference.

[0120] After the GPU generates multi-viewpoint images of each image (also called an instance) through the viewpoint generation model, the multi-viewpoint image formation results of the multiple images are input into the queue for output. In addition, the performance parameters of model inference in the GPU can also be generated and fed back in real time, so that the GPU can perform real-time allocation management of the resources therein.

[0121] Through the technical solution of the embodiments of the present application, the GPU can manage the processes therein in real time according to the current number of tasks (service requests), and a single process can execute multiple tasks in parallel, thereby being able to optimize the resource utilization in the GPU and improve the speed of the GPU for model inference and generating viewpoint images.

[0122] Figure 14 Fig. shows another schematic diagram of the image processing device provided by the embodiments of the present application.

[0123] As Figure 14As shown, the image processing apparatus 100 includes a main processor 120 and a plurality of image processors 110. The main processor 120 is configured to receive a new image processing task and allocate the new image processing task according to the historical image processing tasks to be executed in the plurality of image processors 110 and / or the operating states of the plurality of image processors 110. In this embodiment, the image processing method executed on the image processor may include: the plurality of image processors sending the historical image processing tasks to be processed and / or the operating states to the main processor, so that the main processor allocates a new image processing task according to the historical image processing tasks to be processed in the plurality of image processors and / or the operating states of the plurality of image processors.

[0124] In an embodiment of the present application, the image processing apparatus 100 may include a plurality of GPUs or a plurality of graphics cards. The CPU may be used to manage the operation of the plurality of GPUs. In some examples, when the CPU receives a new image processing task, it may first evaluate the number of image processing tasks to be processed in the current queue of each GPU, obtain the GPU with the smallest current task number, and send the new image processing task into the corresponding GPU queue. If the number of image processing tasks in the two GPU queues is equal, the GPU with a lower utilization rate is selected for processing.

[0125] The technical solution of the embodiment of the present application can achieve resource balance of multiple GPUs, ensure that each GPU has a high utilization rate, and thus is beneficial to improving the overall performance of the image processing apparatus.

[0126] Figure 15 Another schematic diagram of the image processing apparatus provided by the embodiment of the present application is shown.

[0127] As Figure 15 shown, the CPU also has an image processing task queue (shown as a request management queue in the figure). The CPU may rearrange a plurality of image processing tasks in the image processing task queue according to the priority of the received new image processing task (service request).

[0128] For example, when the CPU receives a service request sent by a service end (service system), it is first placed in the request management queue, and then the request management queue is reordered according to the request priority level (for example, the requests can be divided into multiple levels such as level 1: urgent, level 2: normal, level 3: delayed), and then the resource allocation of the requests is performed.

[0129] Based on the technical solutions of the above application embodiments, the CPU can receive new service requests in real time and then place them in different positions of the queue according to the priority level of the request parameters. That is, perform priority sorting on the algorithm service processing, and then obtain the one with the smallest number of requests in the current corresponding GPU queue by evaluating the number of requests to be processed in the current queue of each GPU. Send the request to the queue corresponding to that GPU. If the number of requests in the two GPU queues is equal, select the corresponding queue with lower GPU utilization for processing.

[0130] Continue to refer to Figure 15 As shown, in the embodiments of the present application, for the multi-viewpoint image results generated by the GPU, two methods of asynchronous callback and synchronous callback are provided to feedback to the service end.

[0131] For the synchronous callback method, the image output task queue of the multi-viewpoint image generation (shown as the result output queue in the figure) of multiple GPUs feeds each image output task in the image output task queue to the CPU. The CPU is used to compare the identifiers of the image output tasks and the identifiers of the input image processing tasks, and output the multi-viewpoint images carried in the image output tasks to the service system.

[0132] For the asynchronous callback method, each GPU among multiple GPUs can output the generated multi-viewpoint images to the service system.

[0133] In the embodiments of the present application. Synchronous callback can also be called synchronous return. The main implementation method of synchronization is that the user request first enters the input task queue, and then through the execution of algorithms for rearrangement and task dispatching, the monitoring thread monitors the data in the output queue. When there is data in the output queue, the result is obtained and synchronously returned by comparing the input and output task IDs. Asynchronous callback can also be called asynchronous return. Asynchronous return is relatively simple, that is, directly feedback through the algorithm processing process to the service caller.

[0134] Synchronous callback or asynchronous callback can meet different calling methods of users. For example, when the processing time of the algorithm service request is long, the algorithm service processing callback can be performed in an asynchronous manner. When the algorithm processing time is short, the service can be synchronously returned.

[0135] In the technical solutions of the embodiments of the present application, the multi-viewpoint image is synchronously called back to the service system, which is beneficial to the orderly execution and output of multiple image processing tasks in the system, and is convenient for unified management of multiple image processing tasks. The multi-viewpoint image is asynchronously called back to the service system, which is beneficial to accelerating the feedback of the multi-viewpoint image to the service system and users.

[0136] In Figure 15In the illustrated embodiment, the image processing device may include M GPUs, where M can be any positive integer greater than 1. For the relevant technical solutions of each of the M GPUs, reference can be made to the above Figure 13 description of the relevant embodiment shown, and no further elaboration will be provided here.

[0137] In some embodiments of the present application, the CPU is further configured to obtain the number of multiple GPUs for performing multi-viewpoint image generation, and the number of processes of each GPU among the multiple GPUs. The CPU is used to map the process identifier and GPU identifier in each GPU, configure the environment variables in each GPU according to the GPU identifier, and load the viewpoint generation model to generate multi-viewpoint images.

[0138] In this embodiment, resource scaling management technology can be adopted. When the algorithm is deployed, the number of GPUs (or the number of graphics cards) in the image processing device (such as a server) can be automatically monitored to adapt the task processing process. In practical applications, the algorithm can be deployed in the form of a mirror container. During the container startup process, the GPU resources (or graphics card resources) are specified, and the algorithm can automatically create task processing processes through the given resources without modifying and adapting the task code.

[0139] Figure 16 Shows a schematic diagram of a resource scaling management provided by an embodiment of the present application.

[0140] As Figure 16 shown, after parameter loading of the GPU, the number of tasks that a single GPU can parallelize can be obtained. When the algorithm service (such as a service for multi-viewpoint map calculation) is started, the number of GPUs (also referred to as the number of graphics cards) used to execute the algorithm service can be obtained. According to the algorithm service, the task processing processes to be started can be determined, and then the task processing processes are allocated to the corresponding GPUs, and then the environment variables are configured to load the model to implement model inference.

[0141] For example, the 3D multi-viewpoint image generation algorithm can execute a maximum of 4 tasks in parallel on a single GPU. In a server with 8 graphics cards, 4 * 8 task processing processes are required. Through the mapping of the process and the graphics card ID, processes 1-4 are allocated to the first card, processes 4-8 are allocated to the second card, and so on. After obtaining the graphics card ID, through adapting the environment variables, processes 1-4 run on the first card, and then the model loading and inference are started.

[0142] Through this technical solution, during the process of performing resource scaling management, the situation of the currently allocated GPU or graphics card resources can be monitored to perform automatic concurrent expansion and contraction, automatically monitor the current number of GPUs, and automatically calculate the number of model algorithm loading instances and the number of started child processes by evaluating the GPU resource usage situation of the algorithm, thereby realizing the concurrent operation of the algorithm.

[0143] The present application also provides an image processing system. Figure 17 The figure shows a schematic diagram of an image processing system provided by an embodiment of the present application.

[0144] As Figure 17 shown, the image processing system 300 includes an algorithm end 310 and a service end 320. Among them, the algorithm end 310 may include the image processing device 100 in any of the above embodiments. The service end 320 is configured to receive an image input by a service system and transmit the image to the algorithm end 310 to generate a multi-viewpoint image of the image, and the algorithm end 310 is configured to feed back the generated multi-viewpoint image to the service system in the service end.

[0145] In the embodiment of the present application, the algorithm end 310 and the service end 320 may be disposed in the same electronic device. For example, both are disposed in a server. Alternatively, the algorithm end 310 and the service end 320 may also be disposed in different electronic devices. For example, the algorithm end 310 is disposed in a server, and the service end 320 may be disposed in a smart terminal. The embodiment of the present application does not limit the specific types of the electronic devices where the algorithm end 310 and the service end 320 are located.

[0146] The service system in the service end may be a software system in the service end, which can be used to receive user requirements and send corresponding calculation tasks to the algorithm end.

[0147] Based on the above embodiments of each application, the present application relates to an effective method for deploying a 3D multi-viewpoint algorithm, realizing efficient algorithm deployment, realizing high-performance algorithm inference on the basis of optimizing the algorithm itself, and being compatible with different calling methods.

[0148] In the first aspect, it relates to optimizing and merging algorithm models. Through the optimization and merging processing of the algorithm model itself, the data transmission of the model between different hardware such as data transmission between the CPU or GPU is reduced, and the execution efficiency of algorithm parallelism. On the other hand, to improve the efficiency of model inference calls and avoid improving the data transmission efficiency between different hardware, different methods of memory sharing are adopted, including technical methods such as GPU shared memory and locked-page memory.

[0149] In the second aspect, it relates to dynamic batching, multi-instance process inference, and resource balancing. Dynamic batching mainly realizes dynamic batching inference of the model by merging model inference data, and at the same time creates multi-instance inference of the model by creating multiple processes, further improving the execution efficiency of the algorithm. On the other hand, parallel inference is performed on different GPUs to improve the execution efficiency of the service algorithm and meet high concurrency requirements. By designing a dynamic load balancing algorithm, reasonable distribution of service requests is realized, and thus efficient service calculation is realized. On the other hand, request priority inference management is realized through a reasonable queue design.

[0150] Thirdly, it involves synchronous and asynchronous call compatibility and resource scaling management. Synchronous and asynchronous inference meets different call methods of users. That is, when the processing time of algorithm service requests is long, the algorithm service processing can be called back asynchronously. When the algorithm processing time is short, the service returns synchronously. This solution adopts the idea of multi-threaded asynchronous programming to manage queues reasonably and monitor different call methods of compatible services. Resource scaling management means that the algorithm automatically expands and contracts concurrently by monitoring the resources obtained currently. It automatically monitors the current number of GPUs, calculates the number of model algorithm loading instances and started child processes automatically by evaluating the GPU resource usage of the algorithm, so as to realize the concurrent operation of the algorithm.

[0151] This application realizes the concurrent management of requests by designing a request scheduling strategy, and realizes distributed inference calculation of algorithm services on multiple graphics cards. At the same time, a reasonable request concurrency strategy is designed to achieve load balancing of each card of the algorithm. On the other hand, through the idea of multi-threaded asynchronous programming, the algorithm service is designed to synchronously and asynchronously return the algorithm processing results to realize the flexible adaptation of the algorithm service. In this solution, the quantization acceleration of the 3D 9-viewpoint basic model is realized, and at the same time, multiple interrelated models are merged to reduce the data transmission between different CPUs and GPUs when the model is called, improve the algorithm calculation efficiency. At the same time, to improve the model inference efficiency, the model inference efficiency is improved by realizing multi-process calls, model inference merging, shared memory and other methods. Secondly, to meet the inference requests on different GPU cards, a reasonable load balancing algorithm is designed to realize the reasonable distribution of requests, realize multi-card inference of the algorithm, and further realize the parallel processing ability of algorithm services and improve the concurrent number supported by the algorithm.

[0152] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists, both A and B exist, and B exists.

[0153] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by those skilled in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments, and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects, rather than to describe a specific order or primary-secondary relationship.

[0154] References to "embodiments" in this application mean that specific features, structures, or characteristics described in connection with the embodiments can be included in at least one embodiment of the application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive of other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described in this application can be combined with other embodiments.

[0155] Those of ordinary skill in the art will recognize that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings or direct couplings or communication connections shown or discussed with each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, the functional units in each embodiment of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0159] When the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0160] As described above, the above are only the specific implementation manners of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed by this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that, Including: Inputting an image into a viewpoint generation model; Performing preprocessing and depth estimation on the image based on the viewpoint generation model to generate a multi-viewpoint image of the image.

2. The image processing method according to claim 1, wherein The viewpoint generation model includes a first viewpoint generation sub-model and a second viewpoint generation sub-model; Wherein, the performing preprocessing and depth estimation on the image based on the viewpoint generation model to generate a multi-viewpoint image of the image includes: Performing preprocessing and depth estimation on the image respectively based on the first viewpoint generation sub-model and the second viewpoint generation sub-model to generate a first viewpoint image and a second viewpoint image of the image in parallel; Wherein, the first viewpoint image and the second viewpoint image are respectively a left viewpoint image and a right viewpoint image, or the first viewpoint image and the second viewpoint image are respectively an upper viewpoint image and a lower viewpoint image.

3. The image processing method according to claim 2, wherein Also including: Inputting the first viewpoint image into the first viewpoint generation sub-model; Processing the first viewpoint image based on the first viewpoint generation sub-model to generate a third viewpoint image of the image; Inputting the second viewpoint image into the second viewpoint sub-model; Processing the second viewpoint image based on the second viewpoint generation sub-model to generate a fourth viewpoint image of the image.

4. The image processing method according to claim 1, wherein The inputting an image into a viewpoint generation model includes: Merging multiple images; Inputting the merged multiple images into the viewpoint generation model; The performing preprocessing and depth estimation on the image based on the viewpoint generation model to generate a multi-viewpoint image of the image includes: Based on the viewpoint generation model, performing preprocessing and depth estimation on the merged multiple images in parallel to form a multi-viewpoint image of the multiple images.

5. The image processing method according to claim 1, wherein The performing preprocessing and depth estimation on the image based on the viewpoint generation model to generate a multi-viewpoint image of the image includes: Loading the viewpoint generation model multiple times to perform preprocessing and depth estimation on multiple images in parallel to form a multi-viewpoint image of the multiple images.

6. The image processing method according to any one of claims 1 to 5, characterized in that, Also including: Storing the multi-viewpoint image through shared memory; Feeding back a storage identifier of the multi-viewpoint image in the shared memory to a main processor, so that the main processor obtains the multi-viewpoint image according to the storage identifier and feeds it back to a service system in a service end.

7. The image processing method according to any one of claims 1 to 5, characterized in that, Also including: Copying the multi-viewpoint image to a pinned memory of a main processor in a service end, so that the main processor in the service end feeds back the multi-viewpoint image in the pinned memory to the service system in the service end.

8. The image processing method according to any one of claims 1 to 5, characterized in that, Also including: During the process of generating each viewpoint image in the multi-viewpoint image, synchronously copying each viewpoint image to the shared memory or the pinned memory of the main processor in the service end.

9. The image processing method according to any one of claims 1 to 5, characterized in that Also including: Loading the viewpoint generation model at least once to start at least one process; Obtaining in real time an image processing task to be executed; Managing the started processes according to the image processing task to be executed.

10. The image processing method according to any one of claims 1 to 5, characterized in that, The method is applied to an image processor, and the method also includes: Multiple image processors send historical image processing tasks to be processed and / or operating states to a main processor, so that the main processor allocates new image processing tasks according to the historical image processing tasks to be processed in the multiple image processors and / or the operating states of the multiple image processors.

11. The image processing method according to claim 10, wherein The main processor has an image processing task queue, and the main processor is used to rearrange multiple image processing tasks in the image processing task queue according to the priorities of the received new image processing tasks.

12. The image processing method according to claim 10, wherein It further includes: The multiple image processors generate an image output task queue for multi-viewpoint images; Each image output task in the image output task queue is fed back to the main processor, and the main processor is used to compare the identifiers of the image output tasks and the identifiers of the input image processing tasks, and output the multi-viewpoint images carried in the image output tasks to the service system.

13. The image processing method according to claim 10, wherein It further includes: Each of the multiple image processors outputs the generated multi-viewpoint images to the service system.

14. The image processing method according to claim 10, wherein, The main processor is used to obtain the number of the multiple image processors for performing multi-viewpoint image generation, and the number of processes of each of the multiple image processors; The main processor is used to map the process identifiers and image processor identifiers in each image processor, configure the environment variables in each image processor according to the image processor identifiers, and load the viewpoint generation model to generate the multi-viewpoint images.

15. The image processing method according to any one of claims 1 to 5, characterized in that, The generating the multi-viewpoint images of the image based on the viewpoint generation model includes: Performing a first preprocessing and a third preprocessing on the image based on the viewpoint generation model; Performing depth estimation on the image after the first preprocessing based on the viewpoint generation model, and performing a second preprocessing on the image after the depth estimation; Performing viewpoint generation based on the viewpoint generation model, the image after the second preprocessing, and the image after the third preprocessing to generate the multi-viewpoint images; Wherein, at least one of the first preprocessing, the second preprocessing, and the third preprocessing includes at least one of the following processes: size scaling, normalization, or standardization.

16. An image processing apparatus, characterized in that, It includes: An image processor for performing the image processing method according to any one of claims 1 to 15.

17. The image processing apparatus according to claim 16, wherein It further includes: A main processor connected to at least one of the image processors, and the main processor is used to control at least one of the image processors.

18. An image processing system, characterized in that, It includes: An algorithm end and a service end, the algorithm end includes the image processing device according to claim 16 or 17; The service end is used to receive an image input by a service system, and transmit the image to the algorithm end to generate a multi-viewpoint image of the image, and the algorithm end is used to feed back the generated multi-viewpoint image to the service system in the service end.

Images