Image processing method and device and electronic equipment
By synchronously initializing the CPU engine and asynchronous initializing the GPU engine, the image processing delay and lag problems caused by slow GPU initialization speed are solved, and the image processing speed and user experience are improved. It is especially suitable for scenarios such as virtual backgrounds with high timeliness requirements.
Patent Information
- Application Number
- CN202310551335.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, the slow GPU initialization speed leads to delay and stuttering of the first frame or the first few frames of the image processed, and the user experience is poor, especially in scenarios with high timeliness requirements such as virtual backgrounds.
The synchronous initialization CPU engine and the asynchronous initialization GPU engine are used. If the GPU engine is not initialized, use the CPU engine to perform image processing to ensure that the first frame or the first few frames of images can be processed in time to avoid lag and black screen.
It improves image processing speed and improves user experience, especially in scenarios with high timeliness requirements, avoids the problem of first frame image lag and short-term black screen, and saves system resources.
Smart Images

Figure CN119006260A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device and electronic device. Background Art
[0002] With the development of computer and Internet technologies, there are more and more application scenarios for image processing using computer technology. In addition, many scenarios have strict requirements on the timeliness of image processing and need to display the processed images in time. For example, video special effects such as virtual background are required in video conferencing and online live broadcasts, and the images processed with virtual backgrounds are displayed in real time. Among them, virtual background is a video special effect method that uses the portrait in the picture captured by the camera as the foreground and then replaces the original image background with a customized picture or video.
[0003] Generally, this kind of image processing usually uses GPU (graphics processing unit) to execute the corresponding neural network algorithm for image processing to save computer resources. However, the GPU initialization speed is relatively slow, which may cause the first frame image processing speed to slow down, resulting in delays and freezes in the first frame or the first few frames of images when processing virtual backgrounds, etc. When using it, users will experience screen freezes and a long black screen time when the camera is turned on.
[0004] Therefore, how to provide a solution to improve the image processing speed and ensure the image processing quality is a technical problem that needs to be solved urgently in this field. Summary of the invention
[0005] The present application provides an image processing method, apparatus, device, storage medium, and computer program product, which can improve the speed of first-frame image processing and avoid problems such as first-frame image freeze.
[0006] In one aspect, the present application provides an image processing method, the method comprising: receiving a first image to be processed; Determining whether the graphics processor engine has been initialized, wherein the initialization includes loading a pre-created image processing model; If it is determined that the initialization of the graphics processor engine is not completed, the central processing unit engine is used to perform image processing on the first image to be processed according to the image processing model; if it is determined that the initialization of the graphics processor engine is completed, the graphics processor engine is used to perform image processing on the first image to be processed according to the image processing model; wherein the graphics processor engine is pre-set to asynchronous thread initialization, and the central processing unit engine is pre-set to synchronous thread initialization.
[0007] Another aspect provides an image processing device, the device comprising: An image receiving module, used for receiving a first image to be processed; An engine selection module, used to determine whether the graphics processor engine has been initialized, wherein the initialization includes loading a pre-created image processing model; An image processing module, configured to, if it is determined that the initialization of the graphics processor engine is not completed, use the central processing unit engine to perform image processing on the first image to be processed according to the image processing model; if it is determined that the initialization of the graphics processor engine is completed, use the graphics processor engine to perform image processing on the first image to be processed according to the image processing model; wherein the graphics processor engine is pre-set to asynchronous thread initialization, and the central processing unit engine is pre-set to synchronous thread initialization.
[0008] Another aspect provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement any of the above-mentioned image processing methods.
[0009] On the other hand, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform any of the above-mentioned image processing methods.
[0010] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image processing method provided in the above various optional implementations.
[0011] The image processing method, apparatus, device, storage medium, and computer program product provided by this application have the following technical effects: The image processing method provided by the present application uses a synchronous process to initialize the CPU engine, and at the same time, uses an asynchronous process to initialize the GPU engine. After receiving the image to be processed, it is determined whether the GPU engine has been initialized. If the GPU engine has been initialized, the CPU engine is first used to perform calculations on the image to be processed based on the image processing model loaded during initialization. After determining that the GPU engine has been initialized, the GPU engine is directly used for image processing to save system resources and ensure system performance. Because the CPU engine initializes quickly, when the GPU initialization is not completed, the CPU engine is used for image processing to ensure that the first frame or the first few frames of the image can be processed in a timely manner, and the image processing effects of the previous frames will not be lost, thereby improving the image processing speed. In particular, for scenes with high requirements for image processing timeliness, the problem of the first frame image being stuck or the black screen being black for a short period of time in the early stage can be avoided, thereby improving the user experience. In addition, there is no need to create a callback function, the method is simple, and it is not easy to make mistakes. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0013] Figure 1 is a schematic diagram of an application environment of an image processing method provided in an embodiment of this specification; Figure 2 This is a schematic diagram of the time consumption of the first frame processing using a GPU for virtual background image processing in a video conferencing scenario example of this manual; Figure 3 This is a schematic diagram of the average processing time of using a GPU to process a virtual background image in a video conferencing scenario example of this specification; Figure 4 is a flowchart of an image processing method provided by an embodiment of this specification; Figure 5 is a flowchart of image processing in another embodiment of the present specification; Figure 6 In one embodiment of this specification, Figure 2 Schematic diagram of the first frame processing time after optimizing the virtual background processing of different terminal devices in the video conferencing scenario; Figure 7 is a structural schematic diagram of an image processing device provided in an embodiment of the present application; Figure 8is a block diagram of an electronic device for image processing provided by an embodiment of the present application; Fig. 9 This is a block diagram of another electronic device for image processing provided in an embodiment of the present application. DETAILED DESCRIPTION
[0014] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0015] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0016] See also Figure 1 , Figure 1 1 is a schematic diagram of an application environment of an image processing method provided in an embodiment of this specification, and the application environment may at least include a server 100 and a terminal 200.
[0017] In an optional embodiment, the server 100 can be used to perform image processing. The server 100 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.
[0018] In an optional embodiment, the terminal 200 may be a client that sends image processing results or conference requests to the server 100. Specifically, the terminal 200 may include but is not limited to smart phones, desktop computers, tablet computers, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, vehicle terminals, smart TVs and other types of electronic devices; it may also be software running on the above electronic devices, such as applications, applets, etc. The operating system running on the electronic device in the embodiment of the present application may include but is not limited to Android system, IOS system, Linux, Windows, etc.
[0019] In addition, it should be noted that Figure 1 What is shown is only an application environment of the image processing method, and the embodiments of this specification are not limited to the above.
[0020] In the embodiments of this specification, the server 100 and the terminal 200 may be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitation thereto.
[0021] As people pay more attention to personal privacy, the application scenarios of virtual backgrounds are increasing. For example, virtual background technology may be needed in video conferencing or live webcasts, that is, replacing the background in the image captured by the camera with a specified background or blurring the background, etc., to avoid privacy leakage. The image processing method described in the embodiments of this specification can be mainly applied to the image processing process of the virtual background. Of course, it can also be used in other image processing scenarios that require real-time image processing and have high timeliness requirements, such as some live webcasts that require beautification of characters and real-time display of beautified images.
[0022] Taking virtual background as an example, virtual background has become a rigid requirement in conference scenarios. Its core technology is portrait segmentation based on AI neural network. Based on the frame rate and power consumption requirements of real-time conference user scenarios, the execution of the neural network reasoning process on the terminal should strive to be low-time and low-power. Based on this, GPU reasoning is usually used on some user devices to achieve speed improvement and power reduction. However, the initialization of the GPU takes a certain amount of time, which will cause the time consumption of the first frame to be much greater than the average time consumption of subsequent frames. The user experience is that the picture is stuck and the camera has a long black screen time when it is turned on.
[0023] Figure 2 This is a schematic diagram of the time consumption of the first frame processing using GPU for virtual background image processing in a video conferencing scenario example in this manual. Figure 3FIG. 1 is a schematic diagram of the average processing time of using a GPU to process a virtual background image in a video conferencing scenario example of this manual. Figure 2 , Figure 3 As shown in the figure, the horizontal axis can represent time, and the vertical axis can represent time consumption. Figure 2 , Figure 3 It can be seen that the average time taken to process virtual background images on Windows is about 31.32ms, but the first frame time taken is more than 20 times higher than the average time taken, about 624ms of delay, which is enough for users to feel a noticeable lag. The first frame time taken to process virtual background images on other terminal devices such as Android (Android system), Mac (Macintosh, Apple computer operating system), and iOS (Apple mobile operating system) is also much higher than the average time taken, which will cause noticeable lag in the initial image processing screen.
[0024] In order to avoid the first frame freeze problem of GUP image processing, the initialization of the AI model for image processing can be implemented asynchronously. Developers need to wait for the callback function to return before inputting image data to the AI model for calculation. The process may include: calling the initialization API, passing in parameters and callback functions; when the data to be processed arrives, if the initialization API has not called the callback function to return the initialization result, the data is discarded; if the initialization API has called the callback function and returned the initialization success, the data processing API is called to process the data. It can be seen that its initialization function, initialization callback function, and camera callback function may work in different threads. Developers need to handle thread safety issues by themselves to ensure the timing of object initialization, use, and destruction. This situation will add callback function processing work to developers, which is prone to errors. In addition, when the asynchronous initialization has not succeeded, the camera data cannot be processed, and some of the early image frames will have no special effects. It can be seen that this method may also cause the initial screen of image processing to freeze or be invalid.
[0025] The embodiments of this specification provide an image processing method, which synchronously loads the CPU inference engine and asynchronously loads the GPU inference engine. In the scenario where the GPU inference engine has not been loaded successfully or has failed, the CPU engine is used first to complete the image processing calculation. This can ensure stability and first frame time, improve the speed of image processing, and thus improve the user experience.
[0026] The following describes an image processing method of the present application. Figure 4This is a flowchart of an image processing method provided by an embodiment of this specification. This specification provides method operation steps such as the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only execution order. When the actual system or server product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the embodiment or the method shown in the accompanying drawings. This method can be applied to terminal devices such as computers, tablets, and smart phones, and can also be used in servers. Specifically, Figure 4 As shown, the method may include: S402: Receive a first image to be processed.
[0027] In the specific implementation process, the first image to be processed can be understood as an original image that needs to be processed by virtual background or other image processing, or image data after some preprocessing (such as format conversion, size scaling). The first image to be processed can be a bitmap image, which can also be called a dot matrix image or a raster image, which is mainly composed of single points called pixels (picture elements).
[0028] In the embodiments of this specification, the terminal that performs image processing is generally equipped with a camera, or the server can receive the image captured by the camera of the terminal device, and the image captured by the camera can be used as the first image to be processed. For example: when conducting an online video conference, when the meeting starts, the camera starts and captures images of users participating in the meeting in front of the camera. The image captured by the camera received by the terminal device can be considered as the first image to be processed. The terminal device can process the received first image to be processed according to actual needs, such as virtual background processing or beauty processing or other image processing, and display or send the processed image to the server. Of course, the image captured by the camera can also be used as the first image to be processed after some pre-processing operations, and the embodiments of this specification do not make specific limitations.
[0029] The first image to be processed and the second image to be processed in the following text merely represent the order of image acquisition time. The specific differences are described in the following text and will not be repeated here.
[0030] S404: Determine whether the graphics processor engine has been initialized, where the initialization includes loading a pre-created image processing model.
[0031] In the specific real-time process, the composite engine image processing method is adopted in the embodiments of this specification, that is, the method of using the graphics processing unit engine (i.e., GPU engine) and the central processing unit engine (i.e., CPU (Central Processing Unit) engine) to jointly perform image processing. GPU, also known as display core, visual processor, display chip, is a microprocessor that is specially used to perform image and graphics related calculations on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.). The CPU is the final execution unit for information processing and program running. It can have multiple processing cores and is usually called the brain of the computer. Usually, GPU and CPU are both necessary modules for terminal devices for image processing. After receiving the first image to be processed, it can be determined whether the GPU engine has been initialized, such as: whether the GPU engine has been initialized can be determined based on the initialization state of the GPU.
[0032] The initialization described in the embodiments of this specification may include loading a pre-created image processing model, which may be understood as an intelligent algorithm capable of performing specific image processing, such as a neural network algorithm for performing virtual background. The image processing algorithm may be pre-compiled according to actual needs, and the specific structure and content of the image processing model may be set according to actual needs, which is not specifically limited in the embodiments of this specification.
[0033] Specifically, during the initialization of the GPU engine, a pre-compiled image processing model such as a neural network model can be loaded. Similarly, during the initialization of the CPU engine, a pre-compiled image processing model can also be loaded. During subsequent image processing, the GPU or CPU can perform corresponding computational processing on the image according to the algorithm flow in the image processing model loaded during initialization, such as virtual background image processing, and obtain image processing results, such as virtual background image processing results. The GUP engine and the CPU engine can be understood as a kind of inference engine. The GUP engine and the CPU engine can complete the technical components of the AI model inference calculation according to the calculation flow described by the loaded AI model file, and are mostly presented in the form of SDK.
[0034] In addition, in some embodiments of the present specification, before determining whether the graphics processor engine is initialized, the method further includes: Determine whether the composite engine is initialized. If it is determined that the composite engine is not initialized, use a synchronous thread to initialize the central processing unit engine, start an asynchronous thread, and submit a request to initialize the graphics processing unit engine to the asynchronous thread; wherein the composite engine includes a graphics processing unit engine and a central processing unit engine.
[0035] In the specific implementation process, the embodiment of this specification adopts a composite engine mode for image processing, and the composite engine is a GPU engine and a CPU engine. When the image processing starts, the GPU engine and the CPU engine can be initialized and the image processing model files can be loaded respectively. Figure 5 is a flowchart of image processing in another embodiment of this specification. Figure 5 As shown, generally, when receiving the image to be processed, it can be determined whether the composite engine has started to be initialized. If it is determined that the composite engine has not been initialized, the composite engine can be initialized. For example, the initialization status of the GPU engine and the CPU engine can be obtained respectively. If the initialization status of at least one of the engines is initialization or initialization completed, it means that the composite engine has started to be initialized. Otherwise, it means that the composite engine has not been initialized.
[0036] Generally, a composite engine initialization function can be created, and the composite engine initialization function can be used to define the process of initializing the GPU engine and the CPU engine. In the embodiment of this specification, the composite engine initialization function can be defined as performing synchronous thread initialization on the CPU engine, and at the same time, starting an asynchronous thread, and submitting a task to the asynchronous thread to initialize the GPU engine, thereby realizing synchronous initialization of the CPU engine and asynchronous initialization of the GPU engine. Referring to the records of the above embodiments, the idea of synchronization is that all operations are completed before returning to the user. When the composite engine is initialized, there is no need to provide a callback function. When the composite engine initialization function returns, it can be said that at least the CPU engine initialization is completed. Based on this, it can be determined whether the composite engine is initialized by judging whether the composite engine initialization function returns. If it returns, it is determined that the composite engine has been initialized. If it does not return, the composite engine of the specification is not initialized, and the composite engine needs to be initialized.
[0037] Among them, the synchronous thread can be regarded as a single-threaded operation, which needs to wait for the task to be completed and cannot perform other operations before the task is completed. The asynchronous thread can be understood as not needing to wait for the task to be completed. After submitting the task, it returns directly and can continue to perform other operations.
[0038] It can be seen that the composite engine generally only needs to start initialization when the first frame receives the image to be processed. After receiving the message returned by the composite engine initialization function, it can be said that at least the CPU engine has been initialized and image processing can be performed. After determining that the composite engine has been initialized, that is, after receiving the return of the composite engine initialization function, when receiving other images to be processed, it can directly determine whether the GPU engine has been initialized, and no longer determine whether the composite engine has been initialized, so as to improve the speed of image processing.
[0039] For example: when conducting a video conference, it is necessary to perform virtual background processing on the images captured by the camera. When the user starts the video conference function and selects to start the camera function, after receiving the first frame of the image to be processed captured by the camera, you can first determine whether the composite engine composed of the GPU engine and the CPU engine is initialized. If it is determined that the composite engine is not initialized, then start to synchronously initialize the GPU engine and asynchronously initialize the CPU engine, so that the CPU engine and the GPU engine load the image processing model of the virtual background respectively. After receiving the second frame of the image to be processed, when determining whether the composite engine composed of the GPU engine and the CPU engine is initialized, it is determined that the composite engine has been initialized. Then, when receiving the third frame of the image to be processed, it can be directly determined whether the GPU engine is initialized, and the initialization status of the composite engine is no longer determined.
[0040] The process of composite engine initialization can be summarized as follows: (1) Initialize the CPU engine; (2) Start a thread, referred to as looperThread (this thread continues to wait for task execution); (3) Submit the task of initializing the GPU engine to looperThread, and return without waiting for the task to be completed; (4) During the task of initializing the GPU engine, a variable init_state shared by the calling thread and looperThread is maintained to record the initialization status of the GPU engine (not initialized, initializing, initialization successful, initialization failed).
[0041] Generally, CPU initialization does not require the device initialization logic (such as calling system API, etc.) when the GPU is initialized, and the allocation speed of conventional memory is often greater than that of video memory allocation, and there is no decryption and real-time compilation process of the Kernel code. In other words, the speed of CPU engine initialization will be faster than that of GPU engine initialization. In the embodiment of this specification, the CPU engine is initialized with synchronous threads, and the GPU engine is initialized with asynchronous threads at the same time, and the image processing model is loaded separately. The return of the composite engine initialization function means that the initialization of the synchronous engine (CPU calculation mode) is completed and data calculation can be accepted. At the same time, the thread loaded by the asynchronous engine (GPU calculation mode) has been started. Since the initialization speed of the CPU engine is much faster than that of the GPU engine, it is acceptable to wait synchronously for the CPU engine to be initialized, and it will not wait too long. In this way, when the GPU engine is not initialized, the CPU engine can be used directly for image processing. By taking advantage of the fast loading of the CPU engine, the initialization time can be reduced and the calculation of the early picture frames can be guaranteed, avoiding the occurrence of problems such as the first frame freeze or a short black screen at the beginning of image processing.
[0042] In addition, in some embodiments of the present specification, the callback thread of the camera that captures the first image to be processed is used as the synchronization thread for initializing the central processing unit engine.
[0043] In the specific implementation process, the callback thread of the camera used to collect the image to be processed in the terminal device can be used as the synchronization thread for CPU initialization, wherein the camera callback thread can be understood as a piece of program code that the system automatically calls when the camera state changes. The function corresponding to the callback thread is usually used to monitor the state changes of the camera, such as the connection state, recording state, device state, etc. of the camera, and perform corresponding operations according to these state changes. For example: when the video conference is started, the client will prompt the user whether to allow the camera to be turned on. When the user chooses to turn on the camera, the state of the camera changes, and the callback thread of the camera will be started at this time to collect the image to be processed.
[0044] In the embodiment of this specification, the callback thread of the camera is used as the initialization thread of the CPU engine. When the camera starts to collect the image to be processed, the CPU engine is synchronously initialized and the image processing model is loaded. There is no need to enable other threads, and the progress of image processing is not affected, saving resources occupied by image processing. In addition, the initialization function of the CPU engine and the callback function of the camera work in the same thread, and only one thread needs to be maintained, which is not prone to errors and easy to operate.
[0045] In some embodiments of this specification, the step of determining whether the graphics processor engine has been initialized includes: During the initialization of the graphics processor engine, an initialization status identifier is used to record the initialization status of the graphics processor engine in real time, where the initialization status includes one of: uninitialized, initializing, initialization success, and initialization failure; After receiving the first image to be processed, the initialization state flag of the graphics processor engine is obtained. If the initialization state flag of the graphics processor engine indicates that the initialization is successful, it is determined that the initialization of the graphics processor engine is completed; otherwise, it is determined that the initialization of the graphics processor engine is not completed.
[0046] In the specific implementation process, a variable of the initialization state identifier can be set during the initialization of the GPU engine. The initialization state identifier can be used to record the initialization state of the GPU engine in real time, such as changing the value of the initialization state identifier according to the progress of the GPU engine initialization. After receiving the first image to be processed, the initialization state identifier of the GPU engine can be obtained, and the initialization state of the GPU engine can be determined according to the value of the initialization state identifier. In the embodiment of this specification, the initialization state of the GPU engine may include: one of uninitialized, initializing, successful initialization, and failed initialization. When the value of the initialization state identifier corresponds to successful initialization, it means that the GPU engine initialization is complete, and the GPU engine can be used for image processing. If the value of the initialization state identifier corresponds to one of uninitialized, initializing, and failed initialization, it means that the GPU engine initialization is not complete, and the CPU engine is used for image processing.
[0047] The initialization flag is used to record the initialization status of the GPU engine in real time and monitor the status of the GPU engine, so that image processing can be carried out in time after the GPU engine is initialized. Only when the initialization status flag corresponds to the initialization completion does it mean that the GPU engine initialization is completed. If the GPU engine is not initialized or the initialization fails, it is considered that the initialization is not completed, avoiding the problem that the image processing cannot be carried out normally due to the failure of GPU initialization or the initialization is not fully completed.
[0048] In some embodiments of this specification, obtaining the initialization status identifier of the graphics processor engine includes: During the initialization of the graphics processor engine, the synchronization thread for initializing the central processor engine and the asynchronous thread for initializing the graphics processor engine are used to share the initialization state identifier of the graphics processor engine; After receiving the first image to be processed, the initialization state identifier of the graphics processor engine is obtained by using a synchronization thread.
[0049] In the specific implementation process, during the GPU engine initialization process, the CPU-initialized synchronous thread and the GPU engine-initialized asynchronous thread can be used to share the GPU engine initialization status identifier. In this way, after receiving the first image to be processed, it is possible to quickly determine whether the GPU engine has been initialized based on the initialization status identifier shared by the synchronous thread, thereby determining whether to use the CPU engine or the GPU engine to perform image processing calculations, so as to improve the image processing speed.
[0050] S406: If it is determined that the graphics processor engine initialization is not completed, the central processing unit engine is used to perform image processing on the first image to be processed according to the image processing model; if it is determined that the graphics processor engine initialization is completed, the graphics processor engine is used to perform image processing on the first image to be processed according to the image processing model; wherein the graphics processor engine is pre-set to asynchronous thread initialization, and the central processing unit engine is pre-set to synchronous thread initialization.
[0051] In a specific implementation process, in some embodiments, when a GPU is used for image processing, the process can be summarized as follows: 1. Call the system API (Application Programming Interface) to initialize the GPU device.
[0052] 2. Load the precompiled GPU code library, or the real-time online compiled packaged GPU kernel code. If the solution is real-time online compilation, the GPU kernel needs to go through two steps: decryption and compilation.
[0053] 3. Read in the model file that describes the neural network structure and parameters, and allocate GPU computing memory resources.
[0054] 4. Perform calculations.
[0055] It can be seen that the above steps 1-3 occur when the first frame of data to be processed arrives. At the same time, the code logic executed in steps 1-3 all need to rely on the system API and are inevitably accompanied by IO (Input / Output) operations. Limited by the implementation of the system API and the high latency characteristics of IO devices, developers cannot make further optimizations. Therefore, the time consumed by the first frame is often much longer than the average time consumed by subsequent frames. The user experience is that the screen freezes and the camera has a long black screen time when it is turned on.
[0056] Referring to the records of the above embodiments, the embodiments of this specification adopt a mode of dual-engines of GPU engine and CPU engine for image processing. Generally, the initialization speed of CPU engine is faster than that of GPU engine. In the embodiments of this specification, CPU engine is initialized by synchronous thread, and subsequent data processing is continued after CPU engine initialization is completed. However, generally, the initialization speed of CPU engine is relatively fast, and the waiting time is relatively short, which will not affect the progress of image processing. At the same time, during the initialization of CPU engine, asynchronous thread is used to initialize GPU engine. When the initialization of GPU engine is not completed, CPU engine is directly used for image processing.
[0057] Specifically, after receiving the first image to be processed, it is first determined whether the GPU engine has completed initialization. If it is determined that the GPU engine has not completed initialization, the CPU engine is used to perform image processing on the first image to be processed according to the image processing model loaded therein, and a corresponding image processing result is generated. Among them, the specific content of the image processing can be set according to the actual application needs, such as: virtual background image processing or cutout image processing or beauty image processing, etc., and the embodiments of this specification do not make specific limitations. For example, if it is necessary to perform virtual background processing on the image, the image processing model can select a virtual background image processing model. After determining that the GPU engine has not completed initialization, the CPU engine can perform virtual background image processing on the first image to be processed according to the virtual background image processing model loaded during initialization, blur the background of the first image to be processed or replace the background of the first image to be processed with a specified background image, and generate an image processing result corresponding to the first image to be processed.
[0058] Referring to the records of the above-mentioned embodiment, after determining that the composite engine initialization function returns, when receiving other images to be processed subsequently, it can be determined whether the GPU engine has been initialized. If the GPU engine initialization is not completed, the CPU engine continues to be used for image processing. If it is determined that the GPU engine has been initialized, such as: the value of the initialization state identifier of the GPU engine corresponds to successful initialization, it is determined that the GPU engine has been initialized. At this time, the GPU engine is directly used to process the received images to be processed based on the algorithm flow of the image processing model loaded by initialization, so as to save system resources and improve system performance.
[0059] Although the initialization speed of the GPU engine is slower than that of the CPU engine, the CPU engine has been used for image processing when the GPU engine initialization is not completed, which does not affect the progress of image processing. At the same time, after the GPU engine initialization is completed, the GPU engine is used for image processing to save CPU resources for other services, reduce the power consumption of the terminal device, and improve the performance of the terminal device. Because the GPU engine initialization time will not be too long, even if the first frame or even the first few frames of images are processed by the CPU engine, it will not take up too many resources. The embodiment of this specification uses a combination of the GPU engine and the CPU engine to ensure the speed of image processing, avoid the phenomenon of freezing in the first frame or a short black screen in the early stage, and avoid the problem of subsequent image processing occupying too many system resources and affecting system performance.
[0060] For example: when conducting a video conference, it is necessary to perform virtual background processing on the image captured by the camera. When the user starts the video conference function and selects to start the camera function, after receiving the first frame of the image to be processed (i.e., the first image to be processed described in the previous text) captured by the camera, it can be first determined whether the composite engine composed of the GPU engine and the CPU engine is initialized. If it is determined that the composite engine is not initialized, the CPU engine is synchronously initialized, and the GPU engine is asynchronously initialized, so that the CPU engine and the GPU engine load the image processing model of the virtual background respectively. In other words, after the camera is generally turned on, the CPU engine can be synchronously initialized, and the GPU engine can be asynchronously initialized. Because the CPU engine is initialized faster than the GPU engine, after synchronously waiting for the CPU engine to be initialized, the CPU engine can be directly used to perform virtual background image processing on the first frame of the image to be processed according to the loaded image processing model. After receiving the second frame of the image to be processed (i.e., the first image to be processed described in the previous text), at this time, when the first frame of the image to be processed is already being processed, it is determined that the composite engine has been initialized, and it is directly determined whether the GPU engine is initialized. If it is determined that the GPU engine initialization is not completed, continue to use the CPU engine to perform virtual background image processing on the second frame of image to be processed according to the loaded image processing model. If it is determined that the GPU engine initialization is completed, directly use the GPU engine to perform virtual background image processing on the second frame of image to be processed according to the loaded image processing model.
[0061] In some embodiments of this specification, if it is determined that the graphics processor engine has been initialized, after the graphics processor engine is used to perform image processing on the first image to be processed according to the image processing model, the method further includes: A second image to be processed is received, and a graphics processor engine is used to perform image processing on the second image to be processed according to the image processing model, wherein the acquisition time of the second image to be processed is later than the acquisition time of the first image to be processed.
[0062] In the specific implementation process, if during the processing of a certain frame of the first image to be processed, it is determined that the GPU engine has been initialized, and the GPU engine is used to process the first image to be processed, after receiving other images to be processed, namely the second image to be processed, the GPU engine can be directly used to process the received image to be processed, without having to determine whether the GPU engine has been initialized, so as to save time, improve the image processing speed, and save device system resources. Among them, the first image to be processed and the second image to be processed are only different in acquisition time. The first image to be processed can be understood as the image to be processed by the CPU engine and the first frame of the image to be processed that is processed after the GPU engine is initialized. The second image to be processed can be understood as the second frame of the image to be processed processed after the GPU engine is initialized and the image to be processed acquired thereafter.
[0063] For example: in the above video conferencing scenario example, when the first frame of the image to be processed arrives, the GPU engine and the CPU engine are initialized, and the CPU engine is used to perform virtual background image processing on the first frame of the image to be processed according to the loaded image processing model. When the second frame of the image to be processed arrives, it is determined that the GPU engine initialization is complete, and the GPU engine is used to perform virtual background image processing on the second frame of the image to be processed. After the third and fourth frames of the image to be processed are subsequently received, the GPU engine can be directly used for image processing without judging the initialization status of the GPU engine. Among them, the first and second frames of the image to be processed can be understood as the first image to be processed, and the subsequent third, fourth, and fifth frames of the image to be processed can be understood as the second image to be processed.
[0064] In some embodiments of this specification, the method further includes: If the graphics processing engine fails to perform image processing, the central processing unit engine is used to perform image processing on the first image to be processed or the second image to be processed according to the image processing model.
[0065] In the specific implementation process, referring to the records of the above-mentioned embodiments, after determining that the GPU engine has been initialized, the GPU engine is used to perform image processing on the received first image to be processed or the second image to be processed. Because the CPU has initialized and loaded the image processing model, the GPU reasoning engine works in an independent process. Even if the GPU engine has an abnormality, it can fall back to the CPU engine for reasoning. For example, if the GPU engine image processing fails, the CPU engine can continue to be used to process the received image to be processed, avoiding the problem that the image processing function fails after the GPU engine fails or other reasons cause the image processing to fail, affecting the user's use.
[0066] The image processing method provided in the embodiment of this specification uses a synchronous process to initialize the CPU engine, and at the same time, uses an asynchronous process to initialize the GPU engine. After receiving the image to be processed, it is determined whether the GPU engine has been initialized. If the GPU engine has been initialized, the CPU engine is first used to perform calculations on the image to be processed based on the image processing model loaded during initialization. After determining that the GPU engine has been initialized, the GPU engine is directly used for image processing to save system resources and ensure system performance. Because the CPU engine initializes quickly, when the GPU initialization is not completed, the CPU engine is used for image processing to ensure that the first frame or the first few frames of the image can be processed in a timely manner, and the image processing effects of the previous frames will not be lost, thereby improving the image processing speed. In particular, for scenes with high requirements for image processing timeliness, the problem of the first frame image being stuck or the black screen being on for a short period of time in the early stage can be avoided, thereby improving the user experience. In addition, there is no need to create a callback function, the method is simple, and it is not easy to make mistakes.
[0067] Taking the image processing process of the virtual background as an example, in some embodiments of this specification, the image processing process of the virtual background can be summarized as: input bitmap-pre-processing (such as format conversion, size scaling)-execution of neural network reasoning (obtaining foreground and background masks)-post-processing (blurring the background, replacing the background image). Among them, the step of "executing neural network reasoning" usually abstracts the reasoning of the neural network into processing one or more Tensors according to a certain algorithm flow, and finally calculates an output Tensor. Its processing algorithms for Tensors are mostly convolution, ReLU (positive values retain the original value, negative values return to 0), Scale (multiplication and addition operations on real numbers in Tensor) and other algorithms with parallel characteristics. Based on the "parallel characteristics" in its algorithm flow, GPU is generally used to accelerate the algorithm flow to achieve the purpose of reducing latency, saving CPU resources for other businesses, and reducing power consumption.
[0068] However, referring to the records of the above embodiments, the initialization time of the GPU is relatively long, which will affect the processing speed of the first frame image of the virtual background. In the embodiments of this specification, the composite engine mode is adopted, and the image processing model is loaded in two modes, wherein the CPU mode is synchronous loading, and the advantage of the fast loading of the CPU engine is used to reduce the initialization time and ensure the calculation of the previous picture frames, and the GPU mode is asynchronous loading, and after the loading is completed, it is switched to the GPU engine to process the data. Referring to the records of the above embodiments, the composite engine initialization function can be written according to the composite engine initialization process, the GPU engine and the CPU engine are initialized, and the image processing model algorithm of the virtual background is loaded. After receiving the first image to be processed, the initialization state identifier of the GPU engine, i.e., the marking state of init_state, is checked. If it has not been marked successfully, the data is submitted to the CPU engine for calculation; if init_state is marked as successfully initialized, the data is submitted to the GPU engine for calculation, and the result is waited for. If the GPU engine calculation returns successfully, the data is returned to the upper layer. If the return fails, the data is submitted to the CPU engine for calculation.
[0069] Figure 6 In one embodiment of this specification, Figure 2 Schematic diagram of the first frame processing time after optimizing the virtual background processing of different terminal devices in the video conferencing scenario, such as Figure 6 As shown, the Windows terminal uses the composite engine solution provided in the embodiment of this specification to optimize the first frame, and other terminal devices (such as Android, Mac, iOS) use other methods to optimize the virtual background time-consuming processing, and the specific methods are not described in detail in the embodiment of this specification. Figure 6 and Figure 2 From the first frame time consumption curve of the Windows side, it can be seen that the first frame time consumption of the Windows side using the image processing solution provided in the embodiment of this specification is reduced from 624.5ms to 162.9ms, and the first frame time consumption has been significantly improved. Some of the optimization methods of other clients may also have certain effects, but none of them are as obvious as the effect of this solution. It can be seen that the image processing solution provided in the embodiment of this specification can significantly improve the speed of first frame image processing in scenes with strong timeliness requirements such as virtual backgrounds, and avoid the phenomenon of first frame freeze or camera black screen at the beginning of image processing.
[0070] Based on the image processing method described above, one or more embodiments of this specification also provide a client and a server for image processing. The client and the server may include a device (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0071] It can be seen from the technical solutions provided in the above embodiments of this specification that the embodiments of this specification and this application also provide an image processing device, Figure 7 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application, such as Figure 7 As shown, the above device comprises: The image receiving module 710 is used to receive a first image to be processed; The engine selection module 720 is used to determine whether the graphics processor engine has been initialized, and the initialization includes loading a pre-created image processing model; The image processing module 730 is used to use the central processing unit engine to perform image processing on the first image to be processed according to the image processing model if it is determined that the graphics processor engine initialization is not completed, and to use the graphics processor engine to perform image processing on the first image to be processed according to the image processing model if it is determined that the graphics processor engine initialization is completed; wherein the graphics processor engine is pre-set to asynchronous thread initialization, and the central processing unit engine is pre-set to synchronous thread initialization.
[0072] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be described in detail here. The device in the above embodiment may also include other implementation methods according to the description of the method embodiment, and the specific implementation method can refer to the description of the relevant method embodiment, which will not be repeated here one by one.
[0073] Figure 8 is a block diagram of an electronic device for image processing provided by an embodiment of the present application. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an image processing method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc. Fig. 9 is a block diagram of another electronic device for image processing provided by an embodiment of the present application. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented. Those skilled in the art will understand that Figure 8 or Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the image processing method in the embodiment of the present disclosure.
[0074] In an exemplary embodiment, a computer-readable storage medium is also provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform the image processing method in the embodiment of the present disclosure. In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image processing method provided in the above various optional implementations.
[0075] It is understandable that in the specific implementation of this application, user-related data is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0076] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0077] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0078] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image processing method, It is characterized in that The method comprises: receiving a first image to be processed; Determining whether the graphics processor engine has been initialized, wherein the initialization includes loading a pre-created image processing model; If it is determined that the initialization of the graphics processor engine is not completed, the central processing unit engine is used to perform image processing on the first image to be processed according to the image processing model; if it is determined that the initialization of the graphics processor engine is completed, the graphics processor engine is used to perform image processing on the first image to be processed according to the image processing model; wherein the graphics processor engine is pre-set to asynchronous thread initialization, and the central processing unit engine is pre-set to synchronous thread initialization.
2. The method according to claim 1, It is characterized in that Before determining whether the graphics processor engine is initialized, the method further includes: Determine whether the composite engine is initialized. If it is determined that the composite engine is not initialized, use a synchronous thread to initialize the central processing unit engine, start an asynchronous thread, and submit a request to initialize the graphics processing unit engine to the asynchronous thread; wherein the composite engine includes the graphics processing unit engine and the central processing unit engine.
3. The method according to claim 1, It is characterized in that The callback thread of the camera that captures the first image to be processed is used as a synchronization thread for initializing the central processing unit engine.
4. The method according to claim 1, It is characterized in that If it is determined that the graphics processor engine has been initialized, after the graphics processor engine is used to perform image processing on the first image to be processed according to the image processing model, the method further includes: A second image to be processed is received, and the second image to be processed is processed by the graphics processor engine according to the image processing model, wherein the acquisition time of the second image to be processed is later than the acquisition time of the first image to be processed.
5. The method according to claim 4, It is characterized in that The method further comprises: If the image processing by the graphics processing unit engine fails, the central processing unit engine is used to perform image processing on the first image to be processed or the second image to be processed according to the image processing model.
6. The method according to claim 1, It is characterized in that The step of determining whether the graphics processor engine has been initialized includes: During the initialization of the graphics processor engine, an initialization status identifier is used to record the initialization status of the graphics processor engine in real time, wherein the initialization status includes one of: uninitialized, initializing, initialization success, and initialization failure; After receiving the first image to be processed, an initialization status identifier of the graphics processor engine is obtained. If the initialization status identifier of the graphics processor engine indicates successful initialization, it is determined that the initialization of the graphics processor engine is completed; otherwise, it is determined that the initialization of the graphics processor engine is not completed.
7. The method according to claim 6, It is characterized in that The obtaining of the initialization status identifier of the graphics processor engine includes: During the initialization process of the graphics processor engine, the initialization state identifier of the graphics processor engine is shared by the synchronous thread for initializing the central processing unit engine and the asynchronous thread for initializing the graphics processor engine; After receiving the first image to be processed, the synchronization thread is used to obtain an initialization state identifier of the graphics processor engine.
8. An image processing device, It is characterized in that The device comprises: An image receiving module, used for receiving a first image to be processed; An engine selection module, used to determine whether the graphics processor engine has been initialized, wherein the initialization includes loading a pre-created image processing model; An image processing module, configured to, if it is determined that the initialization of the graphics processor engine is not completed, use the central processing unit engine to perform image processing on the first image to be processed according to the image processing model; if it is determined that the initialization of the graphics processor engine is completed, use the graphics processor engine to perform image processing on the first image to be processed according to the image processing model; wherein the graphics processor engine is pre-set to asynchronous thread initialization, and the central processing unit engine is pre-set to synchronous thread initialization.
9. An electronic device, It is characterized in that include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method as claimed in any one of claims 1 to 7.
