Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium. The method is applied to a system layer of equipment, and comprises the following steps: acquiring a current video frame in a camera output stream of a hardware abstraction layer; determining a portrait included in the current video frame; and determining a display image according to a preset virtual background image and the portrait, wherein the display image is applied to a process running in an application layer. According to the implementation mode, unsafe factors such as vulnerabilities, unsafe network environments and installed malicious software in a third-party library or service can be avoided, so that a user can better protect own privacy; meanwhile, the method facilitates the unified management of the virtual background, simplifies the operation process, improves the convenience, and reduces the risk of misoperation of the user. In addition, the method also reduces repeated calculation and resource waste, improves the overall performance of the method, and has high expandability and maintainability at the same time.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] For the sake of privacy protection and other needs, virtual backgrounds are often used in various online meetings, live broadcasts, and teaching interaction scenarios.

[0003] In related technologies, the virtual background function usually cannot achieve a good privacy protection effect. For example, the original data stream of the camera may be stolen by the application software, and the user may also cause privacy leakage due to improper settings. In addition, when using some applications that do not provide a virtual background function, users are completely unable to protect their privacy security. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device, electronic device and storage medium.

[0005] A first aspect of the present disclosure provides an image processing method, which is applied to a system layer of a device, and the method includes:

[0006] Get the current video frame in the camera output stream of the hardware abstraction layer;

[0007] Determining a human portrait included in the current video frame;

[0008] A display image is determined according to a preset virtual background image and the portrait, and the display image is applied to a process running at an application layer.

[0009] Optionally, the method further includes:

[0010] The display image is written into a camera preview buffer of the system layer, so that a process running in the application layer obtains the display image based on a camera interface provided by the system layer.

[0011] Optionally, the method is deployed in a camera output stream component at the system layer.

[0012] Optionally, determining the portrait contained in the current video frame includes:

[0013] Determine the portrait included in the current video frame based on the portrait segmentation model;

[0014] The method further comprises:

[0015] When the operating system of the device is started, the network parameter file of the portrait segmentation model is loaded into the system layer, and the inference engine of the portrait segmentation model is initialized.

[0016] Optionally, the portrait segmentation model includes a portrait segmentation model based on a TensorFlowLite model.

[0017] Optionally, an image processing control is configured in the system setting application of the device, wherein the image processing control is used to receive a user instruction indicating a virtual background image;

[0018] The step of determining the display image according to the preset virtual background image and the portrait includes:

[0019] Based on the system property interface, a virtual background image is determined according to the user instruction, and a display image is determined according to the virtual background image and the portrait.

[0020] A second aspect of the present disclosure provides an image processing apparatus, applied to a system layer of a device, the apparatus comprising:

[0021] The frame acquisition module is used to obtain the current video frame in the camera output stream of the hardware abstraction layer;

[0022] A portrait segmentation module, used to determine the portrait contained in the current video frame;

[0023] The background replacement module is used to determine a display image according to a preset virtual background image and the portrait, and the display image is applied to a process running in the application layer.

[0024] Optionally, the device further comprises:

[0025] The writing module is used to write the display image into the camera preview buffer of the system layer, so that the process running in the application layer obtains the display image based on the camera interface provided by the system layer.

[0026] Optionally, the device is deployed in a camera output stream component of the system layer.

[0027] Optionally, when the portrait segmentation module is used to determine the portrait contained in the current video frame, it is specifically used to:

[0028] Determine the portrait included in the current video frame based on the portrait segmentation model;

[0029] The device also includes:

[0030] An initialization module is used to load the network parameter file of the portrait segmentation model to the system layer and initialize the inference engine of the portrait segmentation model when the operating system of the device is started.

[0031] Optionally, the portrait segmentation model includes a portrait segmentation model based on a TensorFlowLite model.

[0032] Optionally, an image processing control is configured in the system setting application of the device, wherein the image processing control is used to receive a user instruction indicating a virtual background image;

[0033] The background replacement module is used when determining the display image according to the preset virtual background image and the portrait, specifically for:

[0034] Based on the system property interface, a virtual background image is determined according to the user instruction, and a display image is determined according to the virtual background image and the portrait.

[0035] A third aspect of the present disclosure provides a computer program product, including a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0036] A fourth aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0037] A fifth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in the first aspect.

[0038] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0039] The disclosed embodiment directly processes the camera's output data stream within the device's system level and generates a background virtualized display image, which is then output to the application layer process as the camera's output image. The above implementation can avoid unsafe factors such as vulnerabilities in third-party libraries or services, unsafe network environments, installed malware, etc., allowing users to better protect their privacy; at the same time, the above method not only facilitates the unified management of virtual backgrounds, but also simplifies the operating process, improves convenience, and reduces the risk of "improper operation" by users; in addition, the above method also reduces repeated calculations and resource waste, improves the overall performance of the method, and is highly scalable and maintainable.

[0040] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0042] Figure 1The following is a flowchart of an image processing method shown in some exemplary embodiments.

[0043] Figure 2 is a schematic diagram of an image synthesis method shown in some exemplary embodiments.

[0044] Figure 3 is an architectural diagram of an image processing method shown in some exemplary embodiments.

[0045] Figure 4 is a block diagram of an image processing apparatus shown in some exemplary embodiments.

[0046] Figure 5 It is a hardware structure diagram of an electronic device shown in some exemplary embodiments. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0048] As mentioned in the background technology, in the related art, the virtual background function usually cannot achieve a good privacy protection effect. For example, some applications may transmit video data to the cloud for processing when processing virtual backgrounds, which increases the risk of data leakage. The virtual background function relies on third-party libraries or services, which is not only inefficient, but also may affect user privacy security if these third-party services have security vulnerabilities. Furthermore, the setting interface and options of each application are different. Users need to spend time learning how to set and adjust the virtual background in each application. Even so, users often make improper settings, mistakenly think that the virtual background is turned on by default, and other factors, which puts them in embarrassing situations, resulting in a poor user experience.

[0049] In view of this, the present disclosure provides an image processing method, device, electronic device and storage medium.

[0050] Next, the embodiments of the present disclosure are described in detail.

[0051] In a first aspect, the present disclosure provides an image processing method, which can be applied to the system layer of an electronic device (for example, in an operating system), wherein the electronic device may include a mobile phone, a tablet computer, a notebook computer, a learning machine (and other devices whose systems are usually specialized). Figure 1 , which may include steps S101 to S103.

[0052] Step S101, obtaining a current video frame in a camera output stream of a hardware abstraction layer.

[0053] In this step, continuous video frames can be obtained by calling corresponding system services (such as calling relevant components in the camera service), setting callback functions (setting callback functions for receiving camera video frames, such as BufferQueueCallback functions), starting preview, and other steps (of course, the above method can also be only for one image frame).

[0054] Optionally, the method is deployed in a camera output stream component of the system layer. In other words, the code for executing the method can be deployed in files related to the camera output stream component, such as a camera output stream file. The camera data stream component is a component used to manage and process data streams of hardware devices and provide standardized and unified interfaces to other system layer components or application layer processes, such as Camera3OutputStream or Camera2OutputStream in Android systems, AVCaptureSession and AVCaptureOutput in iOS systems, Media Foundation and IMFSourceReader in Windows systems, V4L2 and GStreamer in Linux systems, etc.

[0055] Take Camera3OutputStream as an example. Camera3OutputStream is part of the camera framework and is used to process the image stream output by the camera hardware. Deploying the above methods in Camera3OutputStream means that these processing logics are directly performed during the transmission of the camera output stream. The above methods make full use of the particularity of the system layer to efficiently process and manage camera data. This is because during the operation of the system, please refer to Figure 2 , the Camera3OutputStream component can interact with the hardware abstraction layer directly through the framework layer, so this step can be completed efficiently and ensure low-latency transmission of video frames ( Figure 2This is a system architecture that takes an Android system as an example, in which the camera provider interface ICameraProvider is used to manage and control camera devices, and the camera device session interface ICameraDeviceSession is an interface associated with ICameraProvider, which is used to handle the session of a single camera device); at the same time, it is precisely because the above method does not require the introduction of additional new files and new components, but can be directly written and deployed in the original camera output stream file at the system layer, so there is no need to deploy the adaptation logic matching the above method in other files or components, which reduces the development difficulty and improves the execution efficiency. In addition, this also completely isolates the application layer from the original camera output stream, protecting user privacy to the greatest extent.

[0056] Step S102: determining a human portrait contained in the current video frame.

[0057] Among them, the portrait may refer to the area corresponding to the human body in the current video frame. These human bodies may include only the main characters in the picture, or may correspond to all the characters in the picture, and these characters may be inferred based on a preset algorithm or a large model. After the area corresponding to the portrait in the current video frame is determined, the pixel information in the area may be recorded separately (that is, the image in the area is cropped), or a portrait mask may be generated, which is used to indicate the area corresponding to the portrait in the current video frame (for example, 0 and 1 are used to indicate whether the pixel is a pixel corresponding to the portrait).

[0058] In the above steps, the determining of the portrait contained in the current video frame may include determining the portrait contained in the current video frame based on a portrait segmentation model, or may include determining the portrait contained in the current video frame based on a preset portrait segmentation algorithm. Optionally, the portrait segmentation model may include a portrait segmentation model based on a TensorFlow Lite model. This is because TensorFlow Lite has been specially optimized for mobile devices and embedded devices, and has the ability to run efficiently in an environment with limited computing resources. Compared with other models, TensorFlowLite is not easily restricted and affected by negative factors such as shortages of computing resources and storage resources. Its smaller model size and optimized inference speed can also provide lower latency in real-time application scenarios of mobile devices, and has special adaptability to the above scenarios.

[0059] In addition, since the method runs at the system layer of the device, optionally, the method may also include: when the operating system of the device is started, loading the network parameter file of the portrait segmentation model to the system layer, and initializing the inference engine of the portrait segmentation model.

[0060] In other words, when the system is turned on and the operating system starts, the components such as the inference engine and model network parameter file on which the above-mentioned portrait segmentation model depends can begin to initialize, so that the above-mentioned method can respond immediately and start execution when the application layer or other system layer components require the camera video stream. Taking the Android system as an example, when the operating system starts, the pre-stored ".tflite" (suffix) model file is loaded, which contains the pre-trained portrait segmentation model parameters. In addition, the TensorFlow Lite inference engine can also be initialized in the Camera Service (a core service in the Android system that is responsible for managing the device's camera hardware and providing an interface for camera functions. The above-mentioned Camera3OutputStream can be a component in the CameraService in some cases). For example, the following model initialization function written in C++ can exemplarily implement the above process and ensure that the model will not be initialized repeatedly (i.e., only initialized once):

[0061]

[0062] Step S103, determining a display image according to a preset virtual background image and the portrait, wherein the display image is applied to a process running at an application layer.

[0063] The preset virtual background image may be specified by a developer, such as a preset fixed image, a solid color image, transparent pixels for filling non-portrait areas, etc. It may also be specified by a user, for example, an image processing control may be configured in the system setting application of the device, wherein the image processing control is used to receive a user instruction indicating a virtual background image; determining the display image according to the preset virtual background image and the portrait may include: determining the virtual background image according to the user instruction based on a system property interface, and determining the display image according to the virtual background image and the portrait.

[0064] The system settings application may be an application in the system for interacting with the user and receiving configuration information indicated by the user. In the Android system, it may usually exist in the form of a "settings.apk" file. After the system is started, the system settings application usually runs in the application layer of the device. In the above method, the system settings application may be configured with controls (configuration items) corresponding to the virtual background function. The controls (may have multiple) may exist in the form of text editing controls, image selection controls, and various switch buttons, and are used to receive user instructions.

[0065] The above user instructions can be used to indicate a virtual background image, that is, they carry information that enables the method to obtain a virtual background image. For example, the user instruction can first indicate whether to turn on the virtual background function at the system level. When the user does not turn on the virtual background function, the portrait or the current video frame can be directly not processed, and the inference engine and model files initialized at system startup can be released to release system resources as much as possible; when the user instruction indicates to turn on the virtual background function, the instruction can be used to indicate the file path of the image, or it can directly indicate an image.

[0066] The user instructions received by the system settings application can be mapped to the components of the execution method in a variety of ways, such as in Figure 2 In the example, Camera3OutputStream can obtain the virtual background image information specified by the user in the system settings application based on the SystemProperties feature (i.e., the system properties interface). Based on this, the above method can interact with the user and execute based on the user's instructions without introducing other application software and deleting or modifying the operating system files, which not only has high efficiency and reliability, but also ensures the simplicity of the system.

[0067] After obtaining the virtual background image based on the above process, see Figure 3 , the virtual background image and the portrait can be synthesized and drawn to obtain a display image, which can be used to flow to the application layer process for its use. Exemplarily, this process can be completed by Open CV (open source computer vision library), thereby maintaining low cost and high maintainability. In this process, more modifications can be made to the portrait, such as adding beauty effects, clothing replacement effects, zoom effects, etc., which will not be repeated here.

[0068] Optionally, the method may further include: writing the display image into a camera preview buffer of the system layer, so that a process running in the application layer obtains the display image based on a camera interface provided by the system layer.

[0069] The camera preview buffer is a memory area used by the camera system to store preview image frames, where the stored image frames can be quickly provided to the application layer for display or processing. It can be a Surface (display surface) or a BufferQueue (buffer queue), etc. Taking the Android system's Camera2 API (i.e., camera interface) as an example, the application layer can set a Surface as a preview target through the CameraCaptureSession component in the Camera2API. The image frames captured by the camera will be written to this Surface by the Camera3OutputStream component. In the above method, when a display image is obtained, the display image can be written to the Surface as the output image of the camera, and the relevant application is notified to read the Surface through the relevant callback function. This approach can make the upstream and downstream components of the method completely unaware of the above method (that is, the logic of the above method is completed by the camera output stream component, and the logic of the application layer receiving the display image and the hardware abstraction layer providing the camera's original image does not change. Regardless of the user's settings, the application layer does not need to pay attention to this). Therefore, the application layer does not need to make corresponding adaptations for the above method, which greatly improves the maintainability, usability and execution efficiency of the above method.

[0070] The above implementation method directly processes the camera's output data stream and generates a background virtualized display image at the system level of the device, and then outputs the image as the camera's output image to the application layer process. Based on this, at the privacy protection level, the virtual background algorithm no longer depends on the application layer process, so the vulnerabilities, insecure network environment, installed malware and other unsafe factors in the above third-party libraries or services will no longer lead to the leakage of the camera's original video stream, thereby allowing users to better protect their privacy. For example, in all scenarios where the "camera is called by the application process", the system layer can automatically shield the real background in the user's environment to prevent sensitive information from being passed to the application layer and network layer.

[0071] Furthermore, at the user experience level, it can provide a consistent virtual background experience in all applications of the operating system. Whether it is video conferencing, live broadcasting platforms, or other applications that require the use of a camera, a unified display image with background virtualization can be obtained directly at the system level, avoiding the problem of users having to re-set the virtual background when switching between different software, and unifying the originally uneven user experience; in addition, it greatly simplifies the operating process, improves convenience, and reduces the risk of "improper operation" by users; furthermore, for corporate users or organizational users, the above-mentioned image processing method running at the system level can easily realize the unified management of the virtual backgrounds of internal personnel of the enterprise or organization, which helps to simplify the management process and ensure the consistency of the corporate image.

[0072] In addition, at the method operation level, since the image processing method is implemented at the system level, it has higher system permissions and is more convenient for optimizing the system resource usage strategy. Compared with each application implementing the virtual background separately, this centralized processing method can reduce repeated calculations and resource waste (the repeated functions originally located in each application can be concentrated at the system level, and repeated calculations are no longer required when different software call the virtual background function at the same time), and improve the overall performance of the method. Moreover, the close relationship between the above method and the operating system also ensures the high scalability of the method. For example, when updating the large model or the function of the method in the future, the developer can directly use the more powerful function support library provided by the system to efficiently complete the development and directly complete the update of the entire image processing method through the form of system update (these operations are imperceptible to the complicated application layer software, and the displayed image can be directly obtained by the application as a "camera original image", and the application layer software does not need to be adapted accordingly).

[0073] The following is a code segment written in C++ language and applicable to an Android system. The code segment can exemplarily implement some or all of the steps in some embodiments provided in the present disclosure.

[0074] In the first step, the current video frame can be obtained and preprocessed, such as resizing (to improve the compatibility of the method and facilitate subsequent processing), normalization (i.e., converting the pixel values ​​of the image into floating-point numbers within a preset range to facilitate mathematical operations and processing, and the model is also easier to learn and train), and format conversion (to make it suitable for visual libraries such as Open CV). For example, this process can be implemented through the following exemplary function.

[0075]

[0076]

[0077] In the second step, the portrait in the current video frame can be obtained based on the portrait segmentation model. For example, the following exemplary function runInference() can implement the process of preprocessing the input image, performing inference, and generating an output portrait mask. Specifically, you can create an inference session and obtain the input tensor, convert the input image to the format required by the model, run the inference session to process the image, obtain the output tensor and generate a portrait mask of a specific category, and finally convert the mask into an OpenCV Mat object and assign it to outputMask for subsequent processing or display.

[0078]

[0079]

[0080] The third step is to process the image data and then return it to the buffer. For example, the following exemplary function Camera3OutputStream::returnBufferLocked() can obtain image data from the hardware abstraction layer, perform processing such as foreground and background separation and virtual background replacement, and finally return the data to the buffer. Specifically, the buffer can be locked first to access the image data, and the image data can be converted into an OpenCV Mat object, and then the above-mentioned function for determining the portrait mask is called, and the portrait mask is generated and merged with the virtual background. The processed image data can then be converted back to YUV format, written to the buffer, and the timestamp and frame count are updated, ensuring that the image data can be efficiently transmitted back to the system after processing.

[0081]

[0082]

[0083]

[0084] Corresponding to the embodiments of the aforementioned method, the present disclosure also provides embodiments of a device and a terminal to which the device is applied.

[0085] The second aspect of the present disclosure provides an image processing device, which is applied to the system layer of a device. Figure 4 , the device comprises:

[0086] The frame acquisition module 401 is used to acquire the current video frame in the camera output stream of the hardware abstraction layer;

[0087] A portrait segmentation module 402 is used to determine the portrait included in the current video frame;

[0088] The background replacement module 403 is used to determine a display image according to a preset virtual background image and the portrait, and the display image is applied to a process running in the application layer.

[0089] Optionally, the device further comprises:

[0090] The writing module is used to write the display image into the camera preview buffer of the system layer, so that the process running in the application layer obtains the display image based on the camera interface provided by the system layer.

[0091] Optionally, the device is deployed in a camera output stream component of the system layer.

[0092] Optionally, when the portrait segmentation module is used to determine the portrait contained in the current video frame, it is specifically used to:

[0093] Determine the portrait included in the current video frame based on the portrait segmentation model;

[0094] The device also includes:

[0095] An initialization module is used to load the network parameter file of the portrait segmentation model to the system layer and initialize the inference engine of the portrait segmentation model when the operating system of the device is started.

[0096] Optionally, the portrait segmentation model includes a portrait segmentation model based on a TensorFlowLite model.

[0097] Optionally, an image processing control is configured in the system setting application of the device, wherein the image processing control is used to receive a user instruction indicating a virtual background image;

[0098] The background replacement module is used when determining the display image according to the preset virtual background image and the portrait, specifically for:

[0099] Based on the system property interface, a virtual background image is determined according to the user instruction, and a display image is determined according to the virtual background image and the portrait.

[0100] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0101] A third aspect of the present disclosure provides a computer program product, including a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0102] For the device embodiments and computer program product embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. In addition, the device embodiments described above are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying creative labor.

[0103] In a fourth aspect, the embodiments of the image processing apparatus provided by the present disclosure can be applied to electronic devices. Figure 5, which exemplarily shows a hardware schematic diagram of an electronic device. For example, the device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0104] Device 500 may include one or more of the following components: a processing component 501 , a memory 502 , a power component 503 , a multimedia component 504 , an audio component 505 , an input / output (I / O) interface 506 , a sensor component 507 , and a communication component 508 .

[0105] The processing component 501 generally controls the overall operation of the device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 501 may include one or more processors 509 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 501 may include one or more modules to facilitate the interaction between the processing component 501 and other components. For example, the processing component 501 may include a multimedia module to facilitate the interaction between the multimedia component 504 and the processing component 501.

[0106] The memory 502 is configured to store various types of data to support operations on the device 500. Examples of such data include instructions for any application or method operating on the device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0107] Power component 503 provides power to various components of device 500. Power component 503 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0108] The multimedia component 504 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 504 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0109] The audio component 505 is configured to output and / or input audio signals. For example, the audio component 505 includes a microphone (MIC), and when the device 500 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 502 or sent via the communication component 508. In some embodiments, the audio component 505 also includes a speaker for outputting audio signals.

[0110] I / O interface 506 provides an interface between processing component 501 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0111] The sensor assembly 507 includes one or more sensors for providing various aspects of status assessment for the device 500. For example, the sensor assembly 507 can detect the open / closed state of the device 500, the relative positioning of components, such as the display and keypad of the device 500, and the sensor assembly 507 can also detect the position change of the device 500 or a component of the device 500, the presence or absence of user contact with the device 500, the orientation or acceleration / deceleration of the device 500, and the temperature change of the device 500. The sensor assembly 507 can also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 507 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 507 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0112] The communication component 508 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G or 5G or a combination thereof. In an exemplary embodiment, the communication component 508 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 508 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0113] In an exemplary embodiment, the device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the image processing method of the electronic device described above.

[0114] In a fifth aspect, the present disclosure, in an exemplary embodiment, further provides a non-transitory computer-readable storage medium including instructions, such as a memory 502 including instructions, and the instructions can be executed by a processor 509 of a device 500 to complete the image processing method of the electronic device. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0115] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the inventions claimed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not claimed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0117] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0118] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. An image processing method, characterized in that: Applied to the system layer of the device, the method comprises: Get the current video frame in the camera output stream of the hardware abstraction layer; Determining a human portrait included in the current video frame; A display image is determined according to a preset virtual background image and the portrait, and the display image is applied to a process running at an application layer.

2. The image processing method according to claim 1, characterized in that: The method further comprises: The display image is written into a camera preview buffer of the system layer, so that a process running in the application layer obtains the display image based on a camera interface provided by the system layer.

3. The image processing method according to claim 1, characterized in that: The method is deployed in the camera output stream component of the system layer.

4. The image processing method according to claim 1, characterized in that: The determining the portrait contained in the current video frame includes: Determine the portrait included in the current video frame based on the portrait segmentation model; The method further comprises: When the operating system of the device is started, the network parameter file of the portrait segmentation model is loaded into the system layer, and the inference engine of the portrait segmentation model is initialized.

5. The image processing method according to claim 4, characterized in that: The portrait segmentation model includes a portrait segmentation model based on a TensorFlow Lite model.

6. The image processing method according to claim 1, characterized in that: An image processing control is configured in the system setting application of the device, wherein the image processing control is used to receive a user instruction indicating a virtual background image; The step of determining the display image according to the preset virtual background image and the portrait includes: Based on the system property interface, a virtual background image is determined according to the user instruction, and a display image is determined according to the virtual background image and the portrait.

7. An image processing device, characterized in that: Applied to the system layer of the device, the device comprises: The frame acquisition module is used to obtain the current video frame in the camera output stream of the hardware abstraction layer; A portrait segmentation module, used to determine the portrait contained in the current video frame; The background replacement module is used to determine a display image according to a preset virtual background image and the portrait, and the display image is applied to a process running in the application layer.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images