Image processing methods and electronic devices
By employing hardware and algorithm registration techniques, the problem of unclear UV camera images has been solved, achieving clear and aesthetically pleasing fusion of RGB and UV images and high-precision sunscreen detection.
Patent Information
- Application Number
- CN202111032943.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-03
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-09-03
AI Technical Summary
Images captured by UV cameras are unattractive and unclear. Existing technology cannot effectively fuse RGB and UV images, resulting in low accuracy in sunscreen detection.
By using hardware registration and algorithm registration techniques, images from RGB and UV cameras are registered and fused to identify sunscreen in facial regions, with each pixel in the RGB and UV images corresponding to the others.
It achieves clear and aesthetically pleasing fusion of RGB and UV images, improving the accuracy and effectiveness of sunscreen detection.
Smart Images

Figure CN115760931B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and more particularly to an image processing method and an electronic device. Background Technology
[0002] With the rapid development of computer technology, the functions of electronic devices such as smartphones and tablets are becoming increasingly diversified. Among them, an ultraviolet (UV) camera, combined with a red-green-blue (RGB) camera, is being introduced into electronic devices to capture and display the sunscreen applied to a user's face.
[0003] However, due to the ability of objects to absorb and reflect ultraviolet light, UV cameras can only display grayscale images and have a poor ability to reproduce details in the images. Therefore, images captured by UV cameras (also known as UV images) are often unattractive and unclear. Summary of the Invention
[0004] This application provides an image processing method and electronic device that can solve the problem of unattractive and unclear UV images in the prior art.
[0005] In a first aspect, embodiments of this application provide an image processing method, which may include: an electronic device acquiring a first image captured by an RGB camera and a second image captured by a UV camera; the first image and the second image are acquired simultaneously; the electronic device registers the first image and the second image, wherein the pixels of the registered first image and the pixels of the registered second image correspond one-to-one; the electronic device fuses the registered first image and the registered second image to obtain a fused image; and the electronic device displays the fused image.
[0006] The aforementioned electronic device can be implemented individually by an electronic device including red-green-blue (RGB) cameras and ultraviolet (UV) cameras (such as mobile phones, tablets, computers, laptops, smart mirrors, smartwatches, etc.), or by a system consisting of a shooting device and a display device. The method described above, by registering the first and second images, allows the registered first and second images to be fused. The fused image combines the clarity of both the first and second images, resulting in a clearer and more aesthetically pleasing image.
[0007] In conjunction with the first aspect, in some embodiments, the electronic device can identify areas such as faces and face masks in the registered first image. Since the first and second images are registered, the pixels in the first and second images correspond one-to-one. The recognition result of the first image can be mapped onto the second image. Therefore, the recognition result can be applied to the second image to detect sunscreen products within a specific area. The sunscreen product can be sunscreen, sunscreen spray, etc. Sunscreen is used as an example in the relevant description of the embodiments of this application.
[0008] In the solution provided in this application, the electronic device including the RGB camera and the UV camera can be hardware registered before leaving the factory.
[0009] Specifically, based on the parameters of the RGB and UV cameras, such as focal length, resolution, camera position, and rotation direction, the transformation relationship between the output images of the RGB and UV cameras is determined. The output images of the RGB or UV cameras are then transformed through this transformation relationship. The transformed images achieve hardware registration, and the aspect ratio and field of view of the two images after hardware registration are kept consistent, providing a basis for the fusion of the two images.
[0010] In conjunction with the first aspect, in some embodiments, the RGB camera and the UV camera on the electronic device can capture images at the same time.
[0011] In conjunction with the first aspect, in some embodiments, the electronic device can perform algorithmic registration on a first image captured by an RGB camera and a second image captured by a UV camera. This algorithmic registration method may include: the electronic device extracting feature lines from the first and second images respectively; the electronic device matching the feature lines of the first and second images; the electronic device determining transformation parameters for the first and second images based on the matched feature lines; the electronic device transforming the first image using the transformation parameters to obtain a transformed first image, or transforming the second image using the transformation parameters to obtain a transformed second image; and then registering the transformed first image with the second image, or registering the transformed second image with the first image. Algorithmic registration can improve the fusion quality of the first and second images.
[0012] In one possible implementation, the feature line can be an edge line, and the electronic device matches the feature lines of the first image and the feature lines of the second image, including: the electronic device determines the edge line with the closest tilt angle and position in the first image and the second image as the matching edge line.
[0013] In another possible implementation, the feature lines can be facial contours, and the electronic device matches the feature lines of the first image and the feature lines of the second image, including: the electronic device determines the facial contour in the second image whose contour features are closest to the contour features of the facial contour in the first image as the matched facial contour.
[0014] In conjunction with the first aspect, in some embodiments, the electronic device can determine the transformation parameters of the first image and the second image based on the matched feature lines, including: the matched feature lines include a first feature line and a second feature line, the first feature line being located on the first image and the second feature line being located on the second image; the electronic device selects a plurality of first coordinate points on the first feature line; the electronic device selects second coordinate points on the second feature line corresponding to the plurality of first coordinate points respectively; and the electronic device calculates the transformation parameters of the first image and the second image based on the plurality of first coordinate points and the plurality of second coordinate points.
[0015] In conjunction with the first aspect, in some embodiments, the electronic device can fuse the registered first image and the registered second image to obtain a fused image, including: the electronic device identifying a target region in the registered first image; the electronic device performing sunscreen detection on the registered second image to obtain a detection result within the target region; and the electronic device superimposing the detection result within the target region onto the registered first image to obtain the fused image.
[0016] The fused image combines the first and second images, making it clearer and more aesthetically pleasing. Furthermore, the pixels of the registered first and second images correspond one-to-one, allowing them to form a mutual mapping relationship.
[0017] Specifically, the target area obtained based on the first image can be mapped onto the second image, and the detection result of sunscreen in the target area of the second image can be mapped onto the first image.
[0018] In conjunction with the first aspect, in one possible implementation, the detection result may include at least one of the application thickness and effectiveness of the sunscreen corresponding to the pixel.
[0019] In conjunction with the first aspect, in one possible implementation, the pixel value of the first pixel within the target region in the fused image is the pixel value corresponding to the detection result of the first pixel, or the weighted sum of the pixel value corresponding to the detection result of the first pixel and the pixel value of the first pixel in the registered first image; the pixel value of the second pixel outside the target region in the fused image is the pixel value of the second pixel in the registered first image.
[0020] In conjunction with the first aspect, in one possible implementation, the electronic device performs sunscreen detection on the registered second image to obtain a detection result within the target area, including: performing sunscreen detection on the image within the target area of the registered second image to obtain a detection result within the target area.
[0021] In conjunction with the first aspect, in another possible implementation, the electronic device performs sunscreen detection on the registered second image to obtain detection results within the target area, including: performing sunscreen detection on the registered second image to obtain detection results across the entire image; and obtaining the detection results within the target area from the detection results across the entire image.
[0022] In conjunction with the first aspect, in some embodiments, the electronic device can identify a target region in the registered first image, including: the electronic device identifying a face region in the registered first image; the electronic device identifying an eye region, an eyebrow region, and a mouth region in the face; and the electronic device removing the eye region, eyebrow region, and mouth region from the face region to obtain the target region.
[0023] In conjunction with the first aspect, in some embodiments, before the electronic device acquires a first image captured by an RGB camera and a second image captured by a UV camera, the electronic device further includes: in response to a detected first operation, the electronic device displays a user interface and activates the RGB camera and the UV camera, the user interface including a preview area; the electronic device displays the fused image including: the electronic device displays the fused image in the preview area.
[0024] In one possible implementation, the electronic device includes the RGB camera and the UV camera.
[0025] In another possible implementation, the electronic device acquires a first image via an RGB camera and a second image via a UV camera, including: the electronic device receiving the first and second images acquired by a shooting device, the shooting device including an RGB camera and a UV camera, the first image being acquired by the shooting device via the RGB camera, and the second image being acquired by the shooting device via the UV camera.
[0026] Secondly, embodiments of this application provide an electronic device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors are used to invoke the computer instructions to cause the electronic device to execute:
[0027] Acquire a first image captured by an RGB camera and a second image captured by a UV camera; the first and second images are acquired simultaneously.
[0028] The first image and the second image are registered, and the pixels of the registered first image correspond one-to-one with the pixels of the registered second image.
[0029] The registered first image and the registered second image are then fused to obtain the fused image;
[0030] The fused image is displayed.
[0031] In one possible implementation, the electronic device provided by the second aspect includes the RGB camera and the UV camera.
[0032] In another possible implementation, the electronic device provided in the second aspect acquires a first image via an RGB camera and a second image via a UV camera, comprising: the electronic device receiving the first image and the second image acquired by the shooting device, the shooting device including an RGB camera and a UV camera, the first image being acquired by the shooting device via the RGB camera, and the second image being acquired by the shooting device via the UV camera.
[0033] In conjunction with the second aspect, in some embodiments, the one or more processors register the first image and the second image, including performing:
[0034] Extract feature lines from the first and second images respectively;
[0035] Match the feature lines of the first image with the feature lines of the second image;
[0036] The transformation parameters of the first and second images are determined based on the matched feature lines;
[0037] The first image is transformed by transforming the first image using transformation parameters to obtain the transformed first image, or the second image is transformed by transforming the second image using transformation parameters to obtain the transformed second image. The transformed first image is then registered with the second image, or the transformed second image is registered with the first image.
[0038] In one possible implementation, the feature lines can be edge lines, and the one or more processors perform matching of the feature lines of the first image and the feature lines of the second image, including performing:
[0039] Edge lines with the same tilt angle and position in the first and second images are identified as matching edge lines.
[0040] In another possible implementation, the feature lines can be edge lines or facial contours, and the one or more processors perform matching of the feature lines of the first image and the feature lines of the second image, including performing:
[0041] The face contour in the second image whose contour features are closest to those in the first image is identified as the matching face contour.
[0042] In conjunction with the second aspect, in some embodiments, the one or more processors perform transformation parameters for determining the first and second images based on the matched feature lines, including performing:
[0043] The matched feature lines include a first feature line and a second feature line, with the first feature line located on the first image and the second feature line located on the second image;
[0044] Select multiple first coordinate points on the first feature line;
[0045] Select second coordinate points on the second feature line that correspond to the multiple first coordinate points respectively;
[0046] The transformation parameters of the first and second images are calculated based on multiple first coordinate points and multiple second coordinate points.
[0047] In conjunction with the second aspect, in some embodiments, the one or more processors perform the fusion of the registered first image and the registered second image to obtain a fused image, including performing:
[0048] Identify the target region in the first registered image;
[0049] Sunscreen detection is performed on the registered second image to obtain the detection results within the target area;
[0050] The detection results within the target area are superimposed onto the registered first image to obtain the fused image.
[0051] In conjunction with the second aspect, in one possible implementation, the one or more processors perform sunscreen detection on the registered second image to obtain detection results within the target area, including performing:
[0052] Sunscreen detection is performed on the target area in the registered second image to obtain the detection results within the target area.
[0053] In conjunction with the second aspect, in another possible implementation, the one or more processors perform sunscreen detection on the registered second image to obtain detection results within the target area, including performing:
[0054] Sunscreen detection was performed on the registered second image to obtain the detection results across the entire image.
[0055] Obtain the detection results within the target area from the detection results across the entire map.
[0056] In conjunction with the first aspect, in some embodiments, the one or more processors perform the task of identifying a target region in the registered first image, including performing:
[0057] Identify the face region in the first registered image;
[0058] Identify the eye, eyebrow, and mouth regions on a human face;
[0059] Remove the eye, eyebrow, and mouth areas from the face region to obtain the target region.
[0060] In conjunction with the second aspect, in one possible implementation, the detection result includes at least one of the following: the application thickness and effectiveness of the sunscreen corresponding to the pixel.
[0061] In conjunction with the second aspect, in one possible implementation, the pixel value of the first pixel in the target region of the fused image is the pixel value corresponding to the detection result of the first pixel, or it is the weighted sum of the pixel value corresponding to the detection result of the first pixel and the pixel value of the first pixel in the registered first image.
[0062] The pixel value of the second pixel outside the target region in the fused image is the pixel value of the second pixel in the registered first image.
[0063] In conjunction with the second aspect, in some embodiments, before the one or more processors perform the acquisition of a first image captured by the RGB camera and a second image captured by the UV camera, the execution further includes:
[0064] In response to the first detected action, the user interface is displayed, and the RGB and UV cameras are turned on. The user interface includes a preview area.
[0065] The merged image includes:
[0066] The fused image is displayed in the preview area.
[0067] In one possible implementation, the electronic device provided by the second aspect includes a display, wherein the one or more processors perform the display of the fused image, including:
[0068] The merged image is displayed on a monitor.
[0069] In another possible implementation, the electronic device provided by the second aspect includes a communication interface, wherein the one or more processors execute the display of the fused image, including:
[0070] The merged image is sent to the display device via a communication interface, and the display device displays the merged image.
[0071] Thirdly, embodiments of this application provide a chip applied to an electronic device. The chip includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.
[0072] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.
[0073] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.
[0074] It is understood that the electronic device provided in the second aspect, the chip provided in the third aspect, the computer program product provided in the fourth aspect, and the computer storage medium provided in the fifth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description
[0075] Figure 1A A system schematic diagram including a shooting device and a display device is provided for an embodiment of this application;
[0076] Figure 1B A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;
[0077] Figures 1C-1D A schematic diagram illustrating two front-facing cameras provided in an embodiment of this application;
[0078] Figure 1E A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;
[0079] Figures 2A-2D A set of user interface diagrams provided for embodiments of this application;
[0080] Figures 3A-3D Another set of user interface diagrams provided for embodiments of this application;
[0081] Figures 4A-4B A schematic flowchart of a set of image processing methods provided in the embodiments of this application;
[0082] Figure 5 A schematic diagram illustrating a hardware registration method provided in an embodiment of this application;
[0083] Figure 6A A schematic diagram of an algorithm registration method based on edge line detection provided in an embodiment of this application;
[0084] Figure 6B A flowchart illustrating an algorithm registration method based on edge line detection provided in this application embodiment;
[0085] Figures 7A-7B A schematic flowchart of a set of algorithm registration methods based on face contour detection provided in the embodiments of this application;
[0086] Figures 8A-8B A schematic flowchart illustrating a set of image fusion methods provided in embodiments of this application;
[0087] Figure 8C A schematic diagram illustrating the generation of a face mask according to an embodiment of this application;
[0088] Figures 9A-9B This is a schematic diagram illustrating a method for presenting the test results of a set of sunscreens provided in an embodiment of this application. Detailed Implementation
[0089] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
[0090] To facilitate understanding of the embodiments of this application, some terms involved in this application will be introduced below.
[0091] 1. RGB cameras and UV cameras
[0092] RGB cameras, or red-green-blue color cameras, are typically the main cameras of electronic devices and can be equipped with white light flash or dual-tone flash. Images captured by RGB cameras (also known as RGB images) are color images. Furthermore, current algorithms for face detection, feature point localization, and face recognition are generally based on RGB images; therefore, RGB images can accurately detect and identify features such as faces, eyes, and eyebrows.
[0093] A UV camera, or ultraviolet camera, primarily consists of an imaging sensor sensitive to ultraviolet light and a filter coating on the lens. The imaging sensor is composed of an array of photodiodes, which can be complementary metal-oxide-semiconductor (CMOS) diodes. Their function is photoelectric conversion, converting light signals into electrical signals. The filter coating allows only ultraviolet light to pass through the lens and reach the imaging sensor for image formation. Visible light and infrared light outside the ultraviolet band cannot pass through the lens and therefore cannot be imaged on the sensor. Generally, UV cameras produce grayscale images. When the imaging sensor detects ultraviolet light, the imaged area appears white or grayish-white; when no ultraviolet light is detected, the imaged area appears black.
[0094] In this embodiment, the electronic device employs multiple cameras, including at least one RGB camera and at least one UV camera. The RGB camera and UV camera are located on the same side of the electronic device, with a smaller spacing between them being better. Their positional relationship can be vertical or horizontal. Furthermore, the two cameras mentioned above can be front-facing cameras or rear-facing cameras.
[0095] 2. Hardware Registration
[0096] Due to differences in installation location, field of view, and distortion, the coordinates of the same subject in two images captured by RGB and UV cameras may not be at the same coordinate position. To eliminate or reduce the difference in coordinates of the same subject in the two images, hardware registration can be performed.
[0097] Hardware registration refers to the pre-designed positioning of the RGB and UV cameras. The closer the RGB and UV cameras are to each other, the better. Furthermore, based on the parameters of the RGB and UV cameras, such as focal length, resolution, camera position, and rotation direction, the transformation relationship between the output images of the RGB and UV cameras is determined. This transformation relationship is then used to transform the output images from either the RGB or UV camera. The transformed image achieves hardware registration. The aspect ratio and field of view of the two hardware-registered images remain consistent, providing a basis for image fusion. Specific hardware registration methods can be found in the relevant descriptions in the embodiments of the registration method below, and will not be repeated here.
[0098] 3. Algorithm Registration
[0099] Algorithm registration can be performed after hardware registration, or it can be used alone to register two images (i.e., the output images from the RGB camera and the UV camera), that is, to determine the transformation relationship between the output images from the RGB camera and the UV camera.
[0100] Algorithm registration can be implemented based on edge detection algorithms or contour line detection algorithms. Specific methods for algorithm registration can be found in the descriptions of algorithm registration in the embodiments below, and will not be repeated here.
[0101] Due to the ability of objects to absorb and reflect ultraviolet light, UV cameras can only display grayscale images, and their ability to reproduce image details is very poor. Therefore, images captured by UV cameras (also known as UV images) are often unattractive and unclear. Furthermore, because UV images have poor clarity, and existing algorithms for face detection, feature point localization, and face recognition are based on RGB images, they cannot accurately detect and recognize UV images. Therefore, it is impossible to build upper-layer applications based on these algorithms using UV images, such as detecting areas where sunscreen has been applied.
[0102] This application provides an image processing method that registers images of the same scene captured simultaneously by an RGB camera and a UV camera, i.e., RGB images and UV images. The pixels of the registered RGB image and the registered UV image correspond one-to-one, meaning that the pixel coordinates of the same subject are consistent in the RGB image and UV image. Furthermore, the registered RGB image and UV image are fused.
[0103] This method, through the registration of RGB and UV images, enables the fusion of the registered RGB and UV images. The fused image combines clear RGB and UV images, thus becoming clearer and more aesthetically pleasing.
[0104] In some embodiments, one way to fuse the registered RGB image and UV image is to: identify the target region in the registered RGB image, such as a face or a face mask; and perform sunscreen detection on the registered UV image to obtain the detection results within the face mask, such as sunscreen thickness and effectiveness; further, superimpose the detection results within the face mask onto the registered RGB image to obtain the fused image.
[0105] The above embodiments can identify areas such as faces and face masks in the registered RGB images. Since the RGB images and UV images are registered, the pixels on the RGB images and UV images correspond one-to-one. The recognition results of the RGB images are mapped onto the UV images. Therefore, the above recognition results can be applied to the UV images to achieve sunscreen detection in specific areas.
[0106] The apparatus and system involved in the embodiments of this application are described below.
[0107] The image fusion method in this application embodiment can be implemented by an electronic device including an RGB camera and a UV camera individually, or by a system consisting of a shooting device and a display device.
[0108] Figure 1A This is a schematic diagram of a system including a shooting device and a display device, provided for an embodiment of this application.
[0109] like Figure 1A As shown, the system's wireless communication includes a shooting device 11 and a display device 12. The shooting device 11 and the display device 12 can establish a connection and transmit data. This connection can be established through short-range wireless communication such as Bluetooth and WLAN, or through mobile communication, or through a wired connection.
[0110] In one implementation, the capturing device 11 may include an RGB camera 11a, a UV camera 11b, and an ultraviolet flash 11c. When the capturing device 11 detects a user operation triggering the capture, in response to the operation, the capturing device 11 simultaneously activates both the RGB camera 11a and the UV camera 11b; images are captured simultaneously by the RGB camera 11a and the UV camera 11b to obtain RGB and UV images; then the RGB and UV images are registered and fused; the fused image is transmitted to a display device 12 connected to the capturing device 11; the display device 12 receives and displays the fused image.
[0111] The ultraviolet flash 11c is used for ultraviolet supplementary lighting in situations where the ultraviolet intensity is very weak. For example, when the UV camera 11b is capturing images indoors, the ultraviolet flash 11c needs to be turned on for ultraviolet supplementary lighting. Because the ultraviolet intensity is very weak indoors, it is impossible to obtain a sufficiently clear ultraviolet image. Therefore, in indoor scenarios, the ultraviolet flash 11c can be turned on for ultraviolet supplementary lighting to ensure the imaging quality of indoor ultraviolet images. When outdoors during the day, the ultraviolet intensity in sunlight is sufficient, and the ultraviolet flash 11c does not need to be turned on, thus eliminating the need for ultraviolet supplementary lighting.
[0112] In another implementation, the imaging device 11 can transmit RGB and UV images to the display device 12; the display device 12 may include a display 11d, the display device 12 receives the RGB and UV images, and displays the fused image through the display 11d by registering and fusing the RGB and UV images.
[0113] Figure 1B This is a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of this application. The electronic device 100 may be an electronic device or a shooting device 11 including an RGB camera and a UV camera, or it may be the aforementioned display device 12.
[0114] Electronic device 100 can be a portable electronic device, such as a mobile phone, tablet computer, smartwatch, smart mirror, or stand-alone ultraviolet (UV) camera. Exemplary embodiments of the portable electronic device include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touchscreen display and / or touchpad). It should also be understood that, in some embodiments, the electronic device can also be a desktop computer.
[0115] Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, at least two cameras 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0116] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0117] For example, when the electronic device 100 is a smart mirror, the sensor module 180, mobile communication module 150, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, button 190, motor 191, indicator 192, etc. may not be essential components of the smart mirror.
[0118] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.
[0119] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.
[0120] It is understood that the processor 110 may also include an AE system. The AE system may be specifically configured within the ISP. The AE system can be used to automatically adjust exposure parameters. Optionally, the AE system may also be integrated into other processor chips. This application embodiment does not limit this.
[0121] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0122] In some embodiments, the processor 110 may include one or more interfaces. The USB interface 130 is an interface compliant with the USB standard specification, specifically a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used for data transfer between the electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices 100, such as AR devices.
[0123] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 through the power management module 141.
[0124] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.
[0125] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.
[0126] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.
[0127] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.
[0128] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194.
[0129] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.
[0130] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with networks and other devices through wireless communication technology.
[0131] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.
[0132] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light-Emitting Diode (AMOLED), a Flexible Light-Emitting Diode (FLED), Mini LED, Micro LED, Micro-OLED, Quantum Dot Light-Emitting Diodes (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.
[0133] Electronic device 100 can acquire data through ISP, camera 193, video codec, GPU, display 194, and application processor.
[0134] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image or video visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.
[0135] Camera 193 is used to capture still images or videos. An object passes through the lens, generating an optical image that is projected onto a photosensitive element. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image or video signal. The ISP outputs the digital image or video signal to a DSP (Digital Signal Processor) for further processing. The DSP converts the digital image or video signal into standard RGB, YUV, or other image or video signal formats.
[0136] In this embodiment of the application, the electronic device 100 may include at least two cameras 193, and the at least two cameras 193 may include an RGB camera and a UV camera.
[0137] A digital signal processor (DSP) is used to process digital signals. Besides processing digital image or video signals, it can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP is used to perform Fourier transforms on the frequency energy.
[0138] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0139] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.
[0140] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.
[0141] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image and video playback, etc.). The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.).
[0142] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.
[0143] The audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal.
[0144] The loudspeaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals.
[0145] The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals.
[0146] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. Electronic device 100 may be equipped with at least one microphone 170C.
[0147] The 170D headphone jack is used to connect wired headphones.
[0148] The sensor module 180 may include one or more sensors, which may be of the same or different types. Understandably, Figure 1B The sensor module 180 shown is only an exemplary division method, and there may be other division methods, which are not limited in this application.
[0149] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 194. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 may also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities may correspond to different operation commands.
[0150] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for image stabilization.
[0151] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.
[0152] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover.
[0153] The accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic device 100, and can be applied to applications such as screen orientation switching and pedometers.
[0154] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.
[0155] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 100 emits infrared light outward through the LED. The electronic device 100 uses the photodiode to detect infrared reflected light from a nearby object. When sufficient reflected light is detected, it can be determined that an object is near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that no object is near the electronic device 100.
[0156] The 180L ambient light sensor is used to detect ambient light intensity.
[0157] The fingerprint sensor 180H is used to acquire fingerprints.
[0158] The 180J temperature sensor is used to detect temperature.
[0159] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.
[0160] The bone conduction sensor 180M can acquire vibration signals.
[0161] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.
[0162] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.
[0163] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.
[0164] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.
[0165] like Figure 1A , Figure 1C , Figure 1D The diagram shown is a structural schematic of some electronic devices provided in the embodiments of this application.
[0166] like Figure 1C , Figure 1D As shown, the electronic device may include an RGB camera 11a, a UV camera 11b, a UV flash 11c, and a display screen 11d. The RGB camera 11a and the UV camera 11b may be front-facing cameras, i.e., located on the same side as the display screen.
[0167] In another implementation, the RGB camera 11a and UV camera 11b of the electronic device can be rear cameras, i.e., opposite to the display screen.
[0168] like Figure 1A As shown, the shooting device may include an RGB camera 11a and a UV camera 11b, and the display device may include a display screen.
[0169] The smaller the distance d between the RGB and UV cameras, the better; for example, 0 < d ≤ 3 cm. Their positional relationship can be either left-right or top-bottom. Furthermore, the RGB and UV cameras can be either front-facing or rear-facing cameras.
[0170] Figure 1E This is a schematic diagram of the software structure of an electronic device 100 provided in an embodiment of this application.
[0171] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into four layers, from top to bottom: the application layer, the application framework layer, the runtime and system libraries, and the kernel layer.
[0172] The application layer can include a series of application packages.
[0173] like Figure 1E As shown, the application package may include applications (also known as apps) such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, UV camera, video, and SMS.
[0174] The application framework layer provides an Application Programming Interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.
[0175] like Figure 1E As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.
[0176] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.
[0177] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.
[0178] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.
[0179] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).
[0180] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.
[0181] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog-style notifications on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.
[0182] The runtime consists of the core libraries and the virtual machine. The runtime is responsible for system scheduling and management.
[0183] The core library consists of two parts: one part is the functionalities that the programming language (e.g., Java) needs to call, and the other part is the system's core library.
[0184] The application layer and application framework layer run in a virtual machine. The virtual machine executes the programming files (e.g., Java files) of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.
[0185] System libraries can include multiple functional modules. For example: Surface Manager, Media Libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.
[0186] The Surface Manager is used to manage the display subsystem and provides fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications.
[0187] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.
[0188] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0189] A 2D graphics engine is a graphics engine for 2D drawing.
[0190] The kernel layer is the layer between hardware and software. The kernel layer includes at least the display driver, camera driver, audio driver, sensor driver, and virtual card driver.
[0191] The following describes some scenarios provided in this application for accurately detecting and presenting information about sunscreen application.
[0192] In essence, a "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It facilitates the conversion between the internal form of information and a form acceptable to the user. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.
[0193] Figure 2A The user interface 21 shown is an example diagram of the main interface of an electronic device. Figure 2A As shown, the user interface 21 includes icons for multiple applications (e.g., camera icon 210, UV camera icon, weather app icon, calendar app icon, photo album app icon, app store app icon, settings app icon, browser app icon, messaging app icon, dialer app icon, etc.). This application embodiment does not limit the content displayed on the user interface 21.
[0194] 1. UV cameras are a shooting mode used in camera applications. Figures 2A to 2D )
[0195] When electronic device 100 detects a user action (such as a touch / click) on the camera application icon 210, in response to the action, electronic device 100 may display, as shown below. Figure 2B The shooting interface 22 is shown. The shooting interface 22 can be the user interface of the default shooting mode, on which the user can take pictures.
[0196] like Figure 2B As shown, the shooting interface 22 may include a parameter adjustment area 220, a preview area 221, a shooting mode selection area 222, an album shortcut control 2231, a shutter control 2232, and a camera flip control 2233.
[0197] The parameter adjustment area 220 may include a flash button, a settings button, etc. The flash button can turn the flash on or off. The settings button can adjust the shooting parameters. The parameter adjustment area 220 may also include other controls, only examples are given here.
[0198] Preview area 221 can be used to display a preview image. The preview image is a fused image obtained by registering and fusing RGB and UV images captured by the RGB camera and UV camera at the same time.
[0199] The shooting mode selection area 222 can display one or more shooting modes. These shooting modes may include: Night Scene Mode 2221, Smart Portrait Mode 2222, Photo Mode 2223, Video Mode 2224, UV Mode 2225, and more 2226.
[0200] The photo album shortcut control 2231 can be used to open the photo album. In response to a user operation, such as a touch operation, applied to the photo album shortcut control 2231, the electronic device 100 can open the photo album.
[0201] The shutter control 2232 can be used to take photos or record videos. The electronic device 100 can detect user operations on the shutter control 2232, and in response to the operation, the electronic device 100 can save the preview image in the preview area 221. In addition, the electronic device 100 can also display thumbnails of the saved images in the album shortcut control 2231.
[0202] The camera flip control 2233 can be used to flip the camera. When the electronic device 100 detects a user operation, such as a touch operation, on the camera flip control 2233, in response to the operation, the electronic device 100 can flip the camera used for shooting, for example, switching the rear camera to the front camera, or switching the front camera to the rear camera.
[0203] When a user operation is detected that affects the shooting mode, the electronic device 100 can activate the user-selected shooting mode. For example, when a user operation is detected that affects the UV mode 2225, the electronic device 100 can activate the UV mode and display... Figure 2C The shooting interface 23 shown indicates that the RGB camera and UV camera are turned on; images are captured by the RGB camera and UV camera at the same time to obtain RGB image and UV image; then the RGB image and UV image are registered and fused, and then the fused image is displayed in the preview area 231, as shown. Figure 2C As shown.
[0204] Optionally, the fused image can be obtained by superimposing RGB images and UV images at a 50% transparency ratio. It should be understood that RGB images and UV images can also be fused at other suitable ratios, and this application embodiment does not limit this.
[0205] When the electronic device 100 detects a user operation on the shutter control 2332 on the shooting interface 23, in response to the operation, the electronic device 100 can save the currently displayed fused image and also store it in the photo album.
[0206] When a user action is detected on the photo album shortcut control 2331, the electronic device 100 can open the photo album and display the user's most recently saved fused image, for example, displaying... Figure 2D The image browsing interface 24 shown.
[0207] Fusion images, by combining RGB images, are clearer and more aesthetically pleasing. They are also suitable for other algorithms such as face detection and feature point localization, and can be used to build upper-layer applications based on these algorithms.
[0208] 2. The UV camera is used as a standalone application. Figures 3A to 3D )
[0209] Similarly, as Figure 3A The user interface 31 shown indicates that the electronic device 100 can detect user actions (such as touch / click) on the UV camera icon 310. In response to this action, the electronic device 100 can simultaneously activate both the RGB camera and the UV camera, and display... Figure 3B The shooting interface shown is 32 or displayed. Figure 3C The shooting interface 33 is shown. Shooting interface 32 or shooting interface 33 can be the user interface of the default shooting mode of the UV camera, on which the user can take pictures.
[0210] like Figure 3B , Figure 3C As shown, the shooting interfaces 32 and 33 may include parameter adjustment areas 320 and 330, preview areas 321 and 331, shooting mode selection areas 322 and 332, album shortcut controls 3231 and 3331, and shutter controls 3232 and 3332.
[0211] The parameter adjustment areas 320 and 330 may include a flash button, a settings button, etc. The flash button can turn the flash on or off. The settings button can adjust the shooting parameters. The parameter adjustment areas 320 and 330 may also include other controls, only examples are given here.
[0212] Preview areas 321 and 331 can be used to display preview images. The preview image is a fused image obtained by registering and fusing RGB and UV images captured by the RGB camera and UV camera at the same time.
[0213] Shooting mode selection areas 322 and 332 can display one or more shooting modes. These shooting mode options can include... Figure 3B Exemplary UV photography mode 3221, UV video recording mode 3222, and more 3223; may also include Figure 3C Exemplary UV modes 3321 and more 3322.
[0214] When a user operation on the shooting mode is detected, the electronic device 100 can activate the shooting mode selected by the user.
[0215] For example, such as Figure 3B As shown, when a user operation is detected on the UV photography mode 3221, the electronic device 100 can activate the UV photography mode and display... Figure 3B The shooting interface 32 shown means that the RGB camera and the UV camera are turned on at the same time; the RGB camera and the UV camera capture images at the same time to obtain RGB images and UV images respectively; then the RGB images and UV images are registered and fused, and then the fused image is displayed.
[0216] When a user operation is detected on the shutter control 3232 on the shooting interface 23, the electronic device 100 can take a picture based on the current shooting mode. For example, when the user touches or clicks the shutter control 3232, if the current shooting mode is UV photography mode, the electronic device saves the current preview image; if the current shooting mode is UV video recording mode, the electronic device starts recording, and stops recording when the shutter control 3232 is clicked again.
[0217] For example, such as Figure 3C As shown, when a user operation is detected on UV mode 3321, electronic device 100 can activate UV mode and display... Figure 3C The shooting interface 33 shown simultaneously activates both the RGB and UV cameras. In one possible implementation, in response to a short press on the shutter control 3332, the electronic device 100 can take a picture and save the current preview image; in response to a long press on the shutter control 3332, the electronic device 100 can take a picture or record a video and save the captured video.
[0218] Optionally, the fused image can be obtained by superimposing RGB images and UV images at a 50% transparency ratio. It should be understood that RGB images and UV images can also be fused at other suitable ratios, and this application embodiment does not limit this.
[0219] The photo album shortcut controls 3231 and 3331 can be used to open the photo album. For example, in response to a user operation, such as a touch operation, on the photo album shortcut controls 3231 and 3331, the electronic device 100 can open the photo album and display the most recently taken image, such as... Figure 3D The image browsing interface 34 shown.
[0220] The aforementioned user interfaces may also contain more or fewer controls, and this application embodiment does not limit this.
[0221] The image processing methods involved in the embodiments of this application are described below, such as... Figure 4A and Figure 4B The example shown is a flowchart of an image processing method. This method may include, but is not limited to, the following steps:
[0222] S401: The electronic device acquires an RGB image captured by an RGB camera and a UV image captured by a UV camera; the RGB and UV images are acquired simultaneously.
[0223] Alternatively, the RGB and UV cameras can be hardware-registered.
[0224] S402: The electronic device registers the RGB image and the UV image, and the pixels of the registered RGB image correspond one-to-one with the pixels of the registered UV image.
[0225] Here, registration can refer to hardware registration, algorithmic registration, or a combination of both. For example, if an electronic device undergoes hardware registration before leaving the factory, and then algorithmic registration is performed, specifically, the electronic device extracts feature lines from both the RGB and UV images; matches the feature lines of the RGB and UV images; determines the transformation parameters of the RGB and UV images based on the matched feature lines; transforms the RGB image using the transformation parameters to obtain the transformed RGB image, or transforms the UV image using the transformation parameters to obtain the transformed RGB image; the transformed RGB image is then registered with the UV image, and the transformed UV image is registered with the RGB image. The aforementioned feature lines can be edge lines or facial contours. Specific methods for hardware and algorithmic registration can be found in the relevant descriptions in the following examples of registration methods, which will not be repeated here.
[0226] The registration process involves calculating the transformation parameters of the RGB and UV images and then performing image transformation using these parameters. These transformation parameters include transformation parameter M1 for converting from the RGB image to the UV image and / or transformation parameter M2 for converting from the UV image to the RGB image. When the transformation parameter is M1, the RGB image is transformed according to M1, resulting in the transformed RGB image, which is the image registered with the UV image, also called the registered RGB image. In this case, the registered UV image is the original UV image. When the transformation parameter is M2, the UV image is transformed according to M2, resulting in the transformed UV image, which is also the image registered with the RGB image, also called the registered UV image. In this case, the registered RGB image is the original RGB image.
[0227] S403: The electronic device fuses the registered RGB image and the registered UV image to obtain the fused image.
[0228] For specific fusion methods, please refer to the relevant descriptions in the following embodiments of image fusion methods, which will not be repeated here.
[0229] For example, an electronic device identifies the target region in a registered RGB image; the electronic device performs sunscreen detection on a registered UV image to obtain the detection result within the target region; the electronic device overlays the detection result within the target region onto the registered RGB image to obtain a fused image.
[0230] Optionally, the target area can be a face mask, that is, the area obtained by removing the eye area, eyebrow area and mouth area from the face area.
[0231] For example, electronic devices can register RGB and UV images, then directly fuse the registered RGB and UV images to display the fused image. Specifically, the pixel values of the fused image are the weighted sum of the pixel values of the RGB image and the pixel values of the UV image.
[0232] For example, the fused image can be obtained by superimposing an RGB image and a UV image with 50% transparency respectively. The two images can also be fused at other ratios, and this application embodiment does not limit this.
[0233] S404: Electronic devices display the merged image.
[0234] The electronic device can display the fused image on its screen or send the fused image to other devices for display. For example, after the shooting device performs the above steps S401-S403, it sends the fused image to the display device, which then displays the fused image.
[0235] The registration method provided in the embodiments of this application is described below.
[0236] The electronic device 100 provided in this application includes at least one RGB camera and at least one UV camera. To achieve the fusion of the RGB image captured by the RGB camera and the UV image captured by the UV camera, it is necessary to first ensure the synchronization and pixel-level alignment of the RGB and UV images. "Synchronization" means that the two cameras capture images of the same object in the same scene at the same time. "Pixel-level alignment" means that any point in the captured object has the same coordinates in both the RGB and UV images.
[0237] However, due to differences in the installation position, field of view, and distortion of the two cameras, the coordinates of facial landmarks on the acquired RGB and UV images are not at the same coordinate position. For example, suppose there are two facial landmarks on the RGB image (such as the center points of the eyes), with coordinates (x1, y1) and (x2, y2). Both the RGB and corresponding UV images have their origin (0, 0) at the top left corner. Therefore, the coordinates of these two facial landmarks on the UV image are not (x1, y1) and (x2, y2), but rather (x1 + δ). x1 y1+δ y1 ), (x2+δ x2 y2+δ y2 ), where, (δ x1 δ y1 ) and (δ x2 δ y2 ) represent the coordinate offsets of these two points on the UV image relative to the RGB image.
[0238] Since the aforementioned coordinate offset is an unknown quantity, it needs to be registered with the RGB camera and the UV camera to determine it. Registration methods can include hardware registration, algorithmic registration, or a combination of both. These will be described below:
[0239] 1. Hardware Registration
[0240] Hardware registration refers to determining the coordinate offset of a point on the UV image relative to the RGB image based on the physical positions of the two cameras and their intrinsic and extrinsic parameters. Hardware registration can be performed by the electronic device 100 before leaving the factory.
[0241] Specifically, the principle of registration can be summarized as follows: there is a transformation relationship between the coordinate points of the simultaneously captured UV image and RGB image, i.e.
[0242] [x',y']=[x,y]×R+T (1)
[0243] Here, R is a transformation matrix that converts RGB image coordinates to UV image coordinates, T is a bias vector, (x',y') represents the coordinates of a point in the UV image, and (x,y) represents the coordinates of the same point in the RGB image. The purpose of hardware calibration is to calculate R and T to achieve the mapping between RGB and UV images.
[0244] In one implementation, such as Figure 5 As shown, the mobile phone can simultaneously capture calibration images using its RGB and UV cameras to obtain RGB and UV images. Based on the position, focal length, resolution, and other parameters of these two cameras, two key points are taken from the UV and RGB images respectively. The coordinates of these two key points on the RGB image are denoted as A1(x1, y1) and A2(x2, y2) respectively, and the coordinates on the UV image are denoted as B1(x1', y1') and B2(x2', y2') respectively. Substituting the coordinates of the two key points into formula (1) can calculate R and T.
[0245] The hardware registration requirement is that the aspect ratio and field of view of the output images (i.e., RGB images and UV images) of the UV camera and the RGB camera must be consistent.
[0246] When a UV camera and an RGB camera capture images, the aspect ratio and field of view of the captured images may differ. When the aspect ratio and field of view of the RGB and UV images are different, the images need to be cropped to obtain images with the same aspect ratio. The adjustment of the image aspect ratio and field of view can be completed by the electronic device 100 before leaving the factory.
[0247] The hardware calibration process of the RGB and UV cameras on the same electronic device 100 can improve the quality of the image after the fusion of RGB and UV images.
[0248] 2. Algorithm Registration
[0249] After acquiring images using RGB and UV cameras on electronic devices, algorithmic registration can be performed on the RGB and UV images to improve the subsequent fusion quality. The algorithmic registration method can be based on edge detection algorithms or contour line detection algorithms, which are described below:
[0250] Registration based on edge detection algorithm:
[0251] Figure 6A The example shown is a schematic diagram of the algorithm registration process for RGB and UV images based on the edge detection algorithm.
[0252] S601: The electronic device extracts the edge lines of the RGB image and the UV image respectively.
[0253] Edge lines in RGB and UV images can be extracted using edge detection algorithms such as the Sobel operator and the Canny operator.
[0254] like Figure 6B As shown, edge lines A1, A2, and A3 are extracted from the RGB image, and edge lines B1, B2, and B3 are extracted from the UV image.
[0255] S602: Match the edge lines of the RGB image with the edge lines of the UV image.
[0256] Whether the edge lines of two images match can be determined by their tilt angle and position in the images. The position of the edge line in the image can be the location of its geometric center or the intersection of its extension with the image edge. Two edge lines with the same tilt angle and position are corresponding. Specifically, the electronic device 100 can calculate the tilt angle and position of the edge lines in the RGB image and the UV image, and determine the two edge lines with the closest tilt angle and position as matching edge lines.
[0257] S603: Aligns RGB and UV images based on matching edge lines.
[0258] In one possible implementation, the electronic device can select N pairs of matching edge lines from the matching edge lines, where N is a positive integer. For example, it can select 3 pairs of matching edge lines, where edge lines A1, A2, and A3 in the RGB image match edge lines B1, B2, and B3 in the UV image, respectively. Based on the position and tilt angle of each pair of edge lines, the mutual transformation parameters of the RGB image and the UV image can be calculated. The transformation parameters calculated based on the 3 pairs of matching edge lines are averaged to obtain the final transformation parameters.
[0259] In another possible implementation, the electronic device 100 can select or randomly select a set of coordinate points (x1, y1), ..., (x1, y1) along an edge of the RGB image. n ,y n Correspondingly, a set of coordinate points (x1', y1'), ..., (x1', y1') are selected on the matching edge line of the UV image. n ',y n '). i and n are positive integers, i≤n,(x i ,y i ) and (x i ',y i ') Corresponding. Furthermore, the electronic device can calculate the transformation parameters of the RGB image and the UV image based on the two sets of coordinate points mentioned above.
[0260] Registration methods based on contour line algorithms:
[0261] Figure 7A and Figure 7B The example shown is a schematic diagram of the algorithm registration process for RGB and UV images based on the contour detection algorithm.
[0262] S701: Extracts the facial contours from an RGB image.
[0263] Electronic devices can detect facial key points, such as facial contour key points, in RGB images based on existing facial key point detection algorithms, thereby obtaining the facial contour in the RGB image. This facial key point detection algorithm can be an image segmentation model, a semantic segmentation model, or a neural network model used to extract facial contours, etc.
[0264] S702: Extract the facial contours from the UV image.
[0265] Electronic devices use face contour extraction models to extract face contours from UV images. These face contour extraction models can be image segmentation models, semantic segmentation models, or neural network models, etc. Due to accuracy issues, multiple face contours may exist during the extraction process on the UV image.
[0266] S703: Match the face contour of the UV image with the face contour of the RGB image.
[0267] Electronic device 100 can employ a contour feature extraction model to extract contour features from both UV and RGB images of the face. Furthermore, the face contour in the UV image whose contour features most closely match those in the RGB image is identified as the matched face contour. The contour feature extraction model can be an active contour model, also known as a SNAKE model, or a model trained using a convolutional neural network. Contour features can be at least one of the following: area, perimeter, center point, etc.
[0268] S704: Aligns RGB and UV images based on matched face contours.
[0269] The electronic device 100 can calculate the transformation parameters of RGB and UV images, and can perform registration by converting the RGB and UV images according to the transformation parameters.
[0270] Specifically, the electronic device 100 can randomly select a set of coordinate points (x1, y1), ..., (x...) on the outline of the RGB image. n , y n At the same time, select a set of coordinate points (x1', y1'), ..., (x1', y1') on the corresponding contour in the UV image. n',y n The electronic device 100 can calculate the transformation parameters of the RGB image and the UV image based on the two sets of coordinate points mentioned above.
[0271] The transformation parameters in the above registration algorithm include transformation parameter M1 for converting from an RGB image to a UV image and / or transformation parameter M2 for converting from a UV image to an RGB image. Transforming the RGB image according to transformation parameter M1 yields the transformed RGB image, which is the image registered with the UV image, also called the registered RGB image. At this point, the registered UV image is the original UV image. Similarly, transforming the UV image according to transformation parameter M2 yields the transformed UV image, which is the image registered with the RGB image, also called the registered UV image. At this point, the registered RGB image is the original RGB image.
[0272] The image transformations mentioned above can be affine transformations, non-rigid transformations, etc.
[0273] 3. Accurately register RGB and UV images by combining hardware registration and algorithm registration.
[0274] This application provides a method for accurately registering RGB and UV images by combining hardware registration and algorithmic registration. Even after registering RGB and UV cameras using hardware registration on a production line, some discrepancies may still exist. To achieve better fusion and presentation of RGB and UV images, algorithmic registration can be further performed on the aforementioned RGB and UV images.
[0275] In this application, the registered RGB image corresponds one-to-one with the pixels on the UV image.
[0276] The image fusion method provided in the embodiments of this application is described below.
[0277] After registering RGB and UV images, the registered RGB and UV images can be fused to display the fused image. The image fusion method involved in the embodiments of this application is described below, such as... Figure 8A and Figure 8B The flowchart shown may include, but is not limited to, some or all of the following steps:
[0278] S801: Obtain the target region based on the registered RGB image.
[0279] The target area can be a face, cheek, forehead, face mask, or other areas. This application embodiment uses a face mask as an example to illustrate the concept.
[0280] For example, when mapping a finger to an RGB image, the position the finger is pointing at is obtained, and a target region is generated centered on that position. This target region can be a circular, rectangular, or triangular area centered on that position; it can also be the facial region where that position is located. For instance, when the finger points to a position on the forehead, the target region is the area where the forehead is located.
[0281] In one implementation, the target region is a face mask. One implementation of face mask recognition based on RGB images can be as follows: First, the electronic device 100 identifies the face region in the RGB image based on a face detection algorithm; second, the electronic device 100 uses a landmark localization algorithm to identify the facial features in the face region, that is, to identify the eye region, eyebrow region, mouth region, etc. in the face region; furthermore, the eye region, eyebrow region, and mouth region are removed from the face region, and the remaining region is the face mask. Figure 8C The example image shown is a face mask, with the white area representing the face.
[0282] S802: Detect sunscreen on a UV image within the target area to obtain the detection results.
[0283] The principle of sunscreen detection: Because chemical sunscreens absorb ultraviolet (UV) rays, areas on the face where chemical sunscreen has been applied absorb a large amount of UV rays, resulting in less UV radiation reflected to the UV camera. Consequently, these areas appear black in the image. Conversely, areas on the face without chemical sunscreen reflect more UV radiation, appearing white in the image. The thickness of the chemical sunscreen affects the amount of UV radiation reflected to the camera, resulting in different colors in the image. The thicker the chemical sunscreen, the darker the image color.
[0284] Taking the target area as a face mask as an example, in one possible implementation, the electronic device can perform sunscreen detection on the image within the face mask area of the UV image.
[0285] In another possible implementation, the electronic device can first perform full-image sunscreen detection on the aforementioned UV image, and then use the aforementioned face mask to remove errors in the detection results, that is, remove the detection results of areas outside the face mask in the full-image detection results to obtain the detection results within the face mask area.
[0286] Specifically, sunscreen detection is performed on each pixel based on the data of each pixel in the UV image to determine the detection result of each pixel. The detection result may include at least one of the application thickness and effectiveness of the sunscreen corresponding to the pixel, but this application embodiment does not limit this.
[0287] Electronic device 100 can determine the pixels on a UV image where sunscreen has been applied, as well as the application thickness and effectiveness of the sunscreen at each pixel, based on the grayscale values corresponding to each pixel. The application thickness or effectiveness of sunscreen at a given pixel can be obtained based on the pixel value (e.g., grayscale value); the smaller the grayscale value, the greater the thickness. Sunscreen effectiveness includes both valid and invalid applications. If the grayscale value of a pixel is below a first threshold, the sunscreen applied to that pixel is sufficient and considered valid. If the grayscale value of a pixel is above the first threshold but below a second threshold, it indicates that sunscreen has been applied to that pixel, but the amount is insufficient and is considered invalid. If the grayscale value of a pixel is above the second threshold, it indicates that no sunscreen has been applied to that pixel.
[0288] S803: The registered RGB image and the sunscreen detection results are fused to obtain the fused image, as shown in Figure 9A.
[0289] The test results for sunscreen include the application thickness and effectiveness of the sunscreen at each pixel within the target area.
[0290] In one specific implementation of S803, the detection image generated based on the sunscreen detection result is used as the foreground image, and the registered RGB image is used as the background image. The sunscreen detection result is superimposed on the registered RGB image, and the resulting image is the fused image.
[0291] The detection image includes all pixels within the target area. The pixel value of a pixel in the detection image is the pixel value corresponding to the detection result of that pixel, which is obtained from the detection result of the sunscreen. For example, a pixel value of purple represents a pixel where sunscreen has been applied; the thicker the layer, the deeper the purple, and vice versa. A pixel value of green indicates that the sunscreen application is sufficient, while red indicates that the sunscreen application is insufficient.
[0292] Furthermore, the labeling method for the detected image can also include outlining the edge lines obtained by delineating the effective area of the sunscreen.
[0293] The darker the color of the outlined edge, the thicker the sunscreen layer applied in that area.
[0294] The overlay can be implemented in, but is not limited to, the following two ways: Figure 9A ,For example:
[0295] Implementation Method 1
[0296] The detected image can be overlaid on the registered RGB image. In this case, the pixel value of a pixel within the target region (e.g., the face mask) in the resulting fused image is the pixel value corresponding to the detection result of that pixel. The pixel values of pixels outside the face mask in the fused image are the pixel values of that pixel in the registered RGB image.
[0297] Implementation Method Two
[0298] The detected image and the registered RGB image can be fused with different ratios of transparency. At this time, the pixel value of the pixel in the target area (such as the face mask) in the fused image is the weighted sum of the pixel value corresponding to the detection result of the pixel and the pixel value of the pixel in the registered RGB image, as shown in formula (2); the pixel value of the pixel outside the face mask in the fused image is the pixel value of the pixel in the registered RGB image.
[0299] P i,j =a×p1 i,j +b×p2 i,j (2)
[0300] In formula (2), (i,j) are the position coordinates of a pixel within the target region in the fused image, a is the opacity of the detected image of that pixel, and b is the opacity of the registered RGB image. 0 < a < 1, 0 < b < 1, and a + b = 1. i,j p1 represents the pixel value corresponding to the pixel with position coordinates (i,j) within the target region in the fused image. i,j To detect the pixel value corresponding to the pixel with position coordinates (i,j) within the target region of the image; p2 i,j The pixel value corresponding to the pixel with position coordinates (i,j) in the target region of the registered RGB image.
[0301] For example, if the detection result of sunscreen has a transparency of 80%, then the pixel values of the R, G, and B channels of the pixel in the target area (such as facemask) in the fused image are respectively equal to the corresponding pixel value of the detection result of that pixel * 0.2 + the pixel value of that pixel in the registered RGB image * 0.8.
[0302] The following is an example of the fused image provided in this application.
[0303] like Figure 9BAs shown, the fused image represents the thickness of the sunscreen.
[0304] In one implementation, the electronic device 100 can overlay the sunscreen application thickness corresponding to each pixel onto an RGB image for fusion and presentation. The electronic device 100 can determine the sunscreen application thickness on the UV image based on the grayscale value corresponding to each pixel, and using existing technology, mark the sunscreen application thickness of each pixel with different colors. Then, it overlays the colored sunscreen application thickness onto the RGB image to visually present the sunscreen application thickness. The electronic device 100 can also represent the sunscreen application thickness by adjusting the color intensity of the pixels in the RGB image; for example, a lighter color indicates a thinner application thickness at the pixel, and a darker color indicates a thicker application thickness.
[0305] It should be understood that the pixel value of a pixel within the face mask in the fused image is the pixel value corresponding to the thickness of the sunscreen applied to that pixel, or the weighted sum of the pixel value corresponding to the thickness of the sunscreen applied to that pixel and the pixel value of that pixel in the registered RGB image; the pixel value of a pixel outside the face mask in the fused image is the pixel value of the same pixel in the registered RGB image.
[0306] like Figure 9B As shown, the fused image demonstrates the effectiveness of the sunscreen.
[0307] In another implementation, such as Figure 9B As shown, the electronic device 100 can fuse and present the effectiveness of sunscreen application on a pixel within an RGB image. In some embodiments, the electronic device 100 can represent the effectiveness of sunscreen application on a pixel by filling it with different colors or outlining edges with different colors. The effectiveness of sunscreen can include situations such as no sunscreen applied / makeup removed completely, sunscreen applied but not enough (poor sun protection), and sunscreen applied in sufficient quantity (good sun protection). Figure 9B As shown, when a pixel is detected as not having sunscreen applied, no color is marked on the RGB image; when a pixel is detected as having sunscreen applied but not enough (i.e., poor sun protection), a warning color (e.g., red) can be marked; when a pixel is detected as having sufficient sunscreen applied (i.e., good sun protection), a positive color (e.g., green or blue) can be marked. Finally, the electronic device overlays the marked colors onto the RGB image. The pixel values corresponding to the effectiveness of sunscreen application on pixels within the face mask in the specific fused image are similar to the pixel values corresponding to the thickness of sunscreen application on the pixels mentioned above, and will not be elaborated further here.
[0308] This application does not impose any restrictions on the labeling method or color of the test results for sunscreen.
[0309] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” “this,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.
[0310] It should also be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. Furthermore, the terms "first," "second," etc., are used to distinguish different objects, not to describe a specific order. The term "a plurality of" refers to two or more.
[0311] It should also be further understood that the reference to "embodiment" in this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.
[0312] The above description is merely a specific embodiment of this application, but the scope of protection of this invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this invention, and these modifications or substitutions should all be covered within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.
Claims
1. An image processing method, characterized in that, Applied to electronic devices, the method includes: The electronic device acquires a first image captured by an RGB camera and a second image captured by a UV camera; the first image and the second image are acquired simultaneously. The electronic device registers the first image and the second image, and the pixels of the registered first image and the pixels of the registered second image correspond one-to-one. The electronic device fuses the registered first image and the registered second image to obtain a fused image; The electronic device displays the fused image; The electronic device registers the first image and the second image, including: The electronic device uses an edge detection algorithm to extract feature lines from the first image and the second image respectively; the feature lines are edge lines; the edge detection algorithm includes the Sobel operator or the Canny operator; The electronic device identifies the edge lines with the closest tilt angle and position in the first image and the second image as the matching edge lines; The electronic device determines the transformation parameters of the first image and the second image based on the matched feature lines; The electronic device transforms the first image using the transformation parameters to obtain a transformed first image, or transforms the second image using the transformation parameters to obtain a transformed second image. The transformed first image is registered with the second image, and the transformed second image is registered with the first image.
2. The method according to claim 1, characterized in that, The electronic device determines the transformation parameters of the first image and the second image based on the matched feature lines, including: The matched feature lines include a first feature line and a second feature line, wherein the first feature line is located on the first image and the second feature line is located on the second image; The electronic device selects multiple first coordinate points on the first feature line; The electronic device selects a second coordinate point on the second feature line that corresponds to the plurality of first coordinate points respectively; The electronic device calculates the transformation parameters of the first image and the second image based on the plurality of first coordinate points and the plurality of second coordinate points.
3. The method according to claim 1, characterized in that, The electronic device fuses the registered first image and the registered second image to obtain a fused image, including: The electronic device identifies the target region in the registered first image; The electronic device performs sunscreen detection on the registered second image to obtain the detection result within the target area; The electronic device overlays the detection results within the target area onto the registered first image to obtain the fused image.
4. The method according to claim 3, characterized in that, The electronic device performs sunscreen detection on the registered second image to obtain detection results within the target area, including: Sunscreen detection is performed on the image within the target area of the registered second image to obtain the detection result within the target area.
5. The method according to claim 3, characterized in that, The electronic device performs sunscreen detection on the registered second image to obtain detection results within the target area, including: Sunscreen detection is performed on the registered second image to obtain detection results across the entire image area; The detection results within the target area are obtained from the detection results within the entire map area.
6. The method according to any one of claims 3-5, characterized in that, The electronic device identifies the target region in the registered first image, including: The electronic device identifies the face region in the registered first image; The electronic device identifies the eye area, eyebrow area, and mouth area within the facial region; The electronic device removes the eye region, eyebrow region, and mouth region from the face region to obtain the target region.
7. The method according to any one of claims 3-6, characterized in that, The detection results include at least one of the following: the application thickness and effectiveness of the sunscreen corresponding to each pixel.
8. The method according to any one of claims 3-7, characterized in that, The pixel value of the first pixel in the target region of the fused image is the pixel value corresponding to the detection result of the first pixel, or it is the weighted sum of the pixel value corresponding to the detection result of the first pixel and the pixel value of the first pixel in the registered first image; The pixel value of the second pixel point outside the target region in the fused image is the pixel value of the second pixel point in the registered first image.
9. The method according to any one of claims 1-8, characterized in that, Before the electronic device acquires the first image captured by the RGB camera and the second image captured by the UV camera, the method further includes: In response to the detected first operation, the electronic device displays a user interface and turns on the RGB camera and the UV camera, the user interface including a preview area; The electronic device displays the fused image by displaying the fused image in the preview area.
10. An electronic device comprising one or more memories and one or more processors, characterized in that, The one or more memories are used to store data and instructions, and the one or more processors are used to call the data and instructions stored in the memories to execute: Acquire a first image captured by an RGB camera and a second image captured by a UV camera; the first image and the second image are acquired simultaneously. The first image and the second image are registered, and the pixels of the registered first image and the pixels of the registered second image correspond one-to-one. The registered first image and the registered second image are fused to obtain a fused image; Display the fused image; The one or more processors performing the registration of the first image and the second image include performing: Feature lines were extracted from the first image and the second image using an edge detection algorithm. The feature line is an edge line; the edge detection algorithm includes the Sobel operator or the Canny operator; Edge lines with the same tilt angle and position in the first image and the second image are identified as matching edge lines; The transformation parameters of the first image and the second image are determined based on the matched feature lines; The first image is transformed using the transformation parameters to obtain a transformed first image, or the second image is transformed using the transformation parameters to obtain a transformed second image. The transformed first image is registered with the second image, and the transformed second image is registered with the first image.
11. The electronic device according to claim 10, characterized in that, The one or more processors perform the process of determining the transformation parameters of the first image and the second image based on the matched feature lines, including performing: The matched feature lines include a first feature line and a second feature line, wherein the first feature line is located on the first image and the second feature line is located on the second image; Select multiple first coordinate points on the first feature line; Select second coordinate points on the second feature line that correspond to the plurality of first coordinate points respectively; The transformation parameters of the first image and the second image are calculated based on the plurality of first coordinate points and the plurality of second coordinate points.
12. The electronic device according to any one of claims 10-11, characterized in that, The one or more processors perform the step of fusing the registered first image and the registered second image to obtain a fused image, including performing: Identify the target region in the registered first image; Sunscreen detection is performed on the registered second image to obtain the detection results within the target area; The detection results within the target area are superimposed onto the registered first image to obtain the fused image.
13. The electronic device according to claim 12, characterized in that, The one or more processors perform sunscreen detection on the registered second image to obtain detection results within the target area, including performing: Sunscreen detection is performed on the image within the target area of the registered second image to obtain the detection result within the target area.
14. The electronic device according to claim 12, characterized in that, The one or more processors perform sunscreen detection on the registered second image to obtain detection results within the target area, including performing: Sunscreen detection is performed on the registered second image to obtain detection results across the entire image area; The detection results within the target area are obtained from the detection results within the entire map area.
15. The electronic device according to any one of claims 12-14, characterized in that, The one or more processors perform the identification of the target region in the registered first image, including performing: Identify the face region in the registered first image; Identify the eye area, eyebrow area, and mouth area in the face; The target region is obtained by removing the eye region, the eyebrow region, and the mouth region from the face region.
16. The electronic device according to any one of claims 12-15, characterized in that, The detection results include at least one of the following: the application thickness and effectiveness of the sunscreen corresponding to each pixel.
17. The electronic device according to any one of claims 12-16, characterized in that, The pixel value of the first pixel in the target region of the fused image is the pixel value corresponding to the detection result of the first pixel, or it is the weighted sum of the pixel value corresponding to the detection result of the first pixel and the pixel value of the first pixel in the registered first image; The pixel value of the second pixel point outside the target region in the fused image is the pixel value of the second pixel point in the registered first image.
18. The electronic device according to any one of claims 10-17, characterized in that, Before the one or more processors execute the acquisition of the first image captured by the RGB camera and the second image captured by the UV camera, the execution further includes: In response to the detected first operation, a user interface is displayed, and the RGB camera and the UV camera are turned on. The user interface includes a preview area. The fused image is displayed as follows: The fused image is displayed in the preview area.
19. A computer-readable storage medium, characterized in that, include: Computer instructions; when the computer instructions are executed on an electronic device, causing the electronic device to perform the method of any one of claims 1-9.
Citation Information
Patent Citations
Image processing method and device, electronic equipment and scale storage medium
CN113034354A
