Focusing method, electronic device, chip system, storage medium and program product

Abstract

The present application relates to the field of image processing technology, and aims to solve the problem of inaccurate focus frame determination, and provides a focusing method, electronic device, chip system, storage medium and program product. The focusing method is applied to an electronic device, and includes: in response to a first instruction indicating to shoot the current scene, starting the image sensor and depth sensor of the electronic device, acquiring the original image output by the image sensor, and acquiring the depth image corresponding to the original image output by the depth sensor; inputting the original image and the depth image into a trained encoder to obtain a first feature image and a second feature image respectively; inputting the first feature image and the second feature image into a trained decoder, fusing the first feature image and the second feature image, and obtaining a significant area on the original image; focusing on objects in the current scene according to the significant area. Make the significant area in the image appear more accurately, and improve the accuracy of the determined focus frame.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a focusing method, electronic equipment, a chip system, a storage medium, and a program product. Background Art

[0002] Users can take pictures and videos through the camera on the electronic device. The traditional focus system focuses on the central area of ​​the captured picture. However, in some scenes, such as scenes with multiple depths of field, the focus frame determined based on the central area of ​​the picture may include other objects besides the object that the user needs to focus on, resulting in the electronic device being unable to capture the object that the user needs to focus on well. Summary of the invention

[0003] The embodiments of the present application provide a focusing method, an electronic device, a chip system, a storage medium and a program product to solve the problem that the focusing frame determined by the existing focusing system is inaccurate, resulting in the inability to capture the object that the user needs to focus on.

[0004] In a first aspect, an embodiment of the present application provides a focusing method, which is applied to an electronic device, wherein the electronic device includes an image sensor and a depth sensor, and the method includes: in response to a first instruction, starting the image sensor and the depth sensor, wherein the first instruction is used to instruct to shoot a current scene; obtaining an original image output by the image sensor, and obtaining a depth image corresponding to the original image output by the depth sensor; inputting the original image and the depth image into a trained encoder, and based on the parameters of the encoder, performing feature extraction on the original image and the depth image to obtain a first feature image and a second feature image respectively; inputting the first feature image and the second feature image into a trained decoder, and based on the parameters of the decoder, fusing the first feature image and the second feature image to obtain a salient area on the original image; focusing on objects in the current scene according to the salient area.

[0005] In the process of focusing of the electronic device, by performing saliency detection on the original image and the depth image corresponding to the original image, a saliency area is obtained, and then focusing is performed based on the saliency area, which reduces the process of processing the original image (such as converting to RGB format) and improves the detection efficiency. Further, the errors existing in the process of processing the original image are avoided, and the data output by the image sensor and the depth sensor can accurately reflect the information of the current scene being photographed. Therefore, the saliency detection based on the data output by the image sensor and the depth sensor can improve the robustness and accuracy of the saliency area detection result. Further, the first feature image and the second feature image are fused, that is, the feature information of the original image is combined with the feature information in the depth image corresponding to the original image, thereby further improving the accuracy of the image saliency detection result, making the saliency area in the image more accurately appear, providing accurate and useful information for subsequent electronic device focusing and other applications, and improving the accuracy of the determined focus frame.

[0006] In a possible implementation, when the number of salient regions is two or more, focusing on an object in a current scene according to the salient regions includes: obtaining, for each salient region, a saliency statistic, a depth statistic, and a first distance corresponding to the salient region, wherein the first distance is used to indicate a distance between the salient region and a central region of an original image; calculating a confidence level of the salient region according to the saliency statistic, the depth statistic, and the first distance; determining a saliency region with the highest confidence level as a target region; and focusing on the object in the current scene according to the target region.

[0007] The confidence of the salient area is calculated by using the saliency statistics, the depth statistics and the first distance, and then the saliency area with the highest confidence is used as the target area, thereby improving the accuracy of the determined focus frame.

[0008] In a possible implementation, focusing on an object in the current scene according to the target area includes: determining a target focus position according to the target area; and driving a focus motor to move the lens to the target focus position to focus on the object in the current scene.

[0009] The target focus position can be determined through the target area, and then the focus motor can be driven directly according to the target focus position to improve the focusing efficiency.

[0010] In one possible implementation, determining a target focus position based on a target area includes: obtaining a comparison table pre-stored in an electronic device, wherein the comparison table records a depth range and a focus position corresponding to the depth range; and determining the target focus position based on a depth statistic of the target area and the comparison table.

[0011] By pre-storing the comparison table, the electronic device can subsequently directly determine the target focus position according to the comparison table, thereby improving the efficiency of determining the target position and further improving the focusing efficiency.

[0012] In one possible implementation, the method also includes: determining whether the picture of the current scene belongs to a multi-depth scene; when the picture of the current scene is a multi-depth scene, segmenting the foreground and background of the original image to obtain a processed original image, and the background in the processed original image is blocked; then inputting the original image and the depth image into the trained encoder includes: inputting the processed original image and the depth image into the trained encoder.

[0013] By removing the background area in advance, the efficiency and accuracy of subsequent saliency detection can be improved.

[0014] In one possible implementation, determining whether the picture of the current scene belongs to a multiple depth of field scene includes: obtaining the number of depth areas in the depth image, each depth area corresponding to a unique depth range; when the number of depth areas in the depth image is greater than or equal to a preset threshold, determining that the picture of the current scene belongs to a multiple depth of field scene; when the number of depth areas in the depth image is less than the preset threshold, determining that the picture of the current scene does not belong to a multiple depth of field scene.

[0015] By processing the depth image, it can be determined whether the current scene belongs to a multi-depth scene, and the depth image is also used for subsequent saliency detection, thereby improving the utilization rate of the depth image.

[0016] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program, the computer program includes program instructions; the processor is used to call the computer program to execute any of the above focusing methods.

[0017] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, the electronic device including an image sensor and a depth sensor, and the chip system including an interface circuit and a processor; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the chip system responds to a first instruction to start the image sensor and the depth sensor, wherein the first instruction is used to instruct to shoot the current scene; obtain an original image output by the image sensor, and obtain a depth image corresponding to the original image output by the depth sensor; input the original image and the depth image into a trained encoder, and based on the parameters of the encoder, extract features of the original image and the depth image to obtain a first feature image and a second feature image respectively; input the first feature image and the second feature image into a trained decoder, and based on the parameters of the decoder, fuse the first feature image and the second feature image to obtain a salient area on the original image; focus on objects in the current scene according to the salient area.

[0018] In a fourth aspect, an embodiment of the present application provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes any of the above focusing methods.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute any of the above focusing methods.

[0020] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1A A shooting scene provided in an embodiment of the present application.

[0022] Figure 1B A schematic diagram of a preview image provided in an embodiment of the present application.

[0023] Figure 2 A schematic diagram of a focusing process provided in an embodiment of the present application.

[0024] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0025] Figure 4 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application.

[0026] Figure 5A schematic diagram of a focusing method flow chart provided in an embodiment of the present application.

[0027] Figure 6 A schematic flow chart of a method for determining a target area provided in an embodiment of the present application.

[0028] Figure 7 A schematic diagram of a chip system structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In this article, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprising a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0030] In this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0031] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0032] In the embodiments of the present application, the depth of field (DoF) refers to the relatively clear imaging range before and after the focus of the camera or other imaging device. A multi-depth scene can be understood as that within the depth of field range, the objects (for example, people or objects) in the captured picture are at different depth positions and the depth distribution is uneven. In other words, there are multiple objects, and the multiple objects have different distance ranges relative to the camera or other imaging device, and the pictures obtained by shooting the multiple objects belong to a multi-depth scene.

[0033] In such Figure 1AIn the shooting scene shown, a portrait 20 is hung on a wall 10, and a desk 30 is placed in front of the wall 10. A user 40 stands in front of the desk 30 and the portrait 20, and holds a pen 50 facing an electronic device 60 (such as a mobile phone). The user 40 activates the electronic device 60 to shoot. The electronic device 60 presents Figure 1B The preview image 70 shown includes the portrait 20 , the hand of the user 40 , and the screen corresponding to the pen 50 .

[0034] Figure 1B The focus frame 80 presented in the preview image 70 shown includes the pen 50 that the user 40 wants to focus on, as well as the user 40's hand, the portrait 20 and other photographed objects. Since the pen 50 and the portrait 20 are at different depth positions in the preview image 70, that is, the pen 50 and the portrait 20 have different distance ranges relative to the electronic device 60, when the electronic device 60 focuses on the objects in the focus frame 80, it will consider the distance of the portrait 20 relative to the lens and move the lens, and the position to which the lens is moved cannot accurately focus on the pen 50, thereby causing the pen 50 that the user 40 wants to focus on to be out of focus.

[0035] like Figure 2 As shown, an exemplary electronic device 60 is presented Figure 1B The focusing process is shown when previewing the image 70 .

[0036] The electronic device 60 includes an image sensor 601 , an image front-end processing layer 602 , a 3A layer 603 , an image processing engine 604 , an image processing layer 605 , a focus motor 606 and a lens 607 .

[0037] The image sensor 601 generates an image of the subject, and the generated image of the subject is an original image in RAW format. The image front-end processing layer 602 obtains the original image output by the image sensor 601. The image front-end processing layer 602 collects the status information of the original image, and the status information includes but is not limited to: automatic white balance status information, automatic exposure status information and automatic focus status information. The image front-end processor layer 602 transmits the collected status information to the 3A layer 603, and the 3A layer 603 includes an automatic exposure module, an automatic white balance module and an automatic focus module. The autofocus module outputs the center focus frame to the focus motor 606 based on the autofocus status information and its internal autofocus algorithm. The focus motor 606 moves the lens 607 based on the center focus frame to focus on the object in the center focus frame. As shown in FIG. Figure 1B As shown, the central focus frame is the area where the center of the preview image is located, that is, the electronic device 60 focuses on the person portrait 20, the hand 40 and the pen 50.

[0038] The image front-end processing layer 602 transmits the received original image to the image processing engine 604 .

[0039] The image processing engine 604 is used to convert the format of the original image, such as converting the RAW format into the YUV format, or into the HSV format, Lab format, CMY format, YCbCr format, etc. The image processed by the image processing engine 604 can be called a 3-channel image, such as an RGB image. The image processing engine 604 transmits the processed RGB image to the image processing layer 605.

[0040] The image processing layer 605 can perform saliency detection on the RGB image processed by the image processing engine 604 based on its internal saliency detection network, and output the area where the salient subject is located, that is, the saliency area. The image processing layer 605 outputs the output saliency area to the autofocus module, and the autofocus module adjusts the focus frame to the saliency area based on the saliency area. The focus motor re-drives the lens based on the adjusted focus frame to focus on the salient subject.

[0041] Conventional saliency detection algorithms generally regard people as salient subjects, that is, the area where people are located as salient areas, so after the image processing layer 605 performs saliency detection, the electronic device 60 will focus on the person portrait 20. Accordingly, the focus frame in the preview image presented by the electronic device 60 will be the area where the person portrait 20 is located, rather than the area where the pen 50 that the user 40 wants to focus on is located.

[0042] In a multi-depth scene, the central focus frame determined based on the auto-focus algorithm is very likely to include an object that the user 40 does not want to focus on, and the object is very likely to have a different depth position from the object that the user wants to focus on, such as Figure 1BThe focus frame 80 in the preview image shown includes the portrait 20 of the person. Alternatively, the salient subject determined by the saliency detection based on the RGB image may not be the pen 50 that the user 40 wants to focus on, and thus the focus frame determined based on the saliency region includes the portrait 20 of the person instead of the pen 50 that the user 40 wants to focus on. As a result, in a multi-depth scene, the accuracy of the determined focus frame is low, and thus when focusing based on the focus frame, the subject that the user wants to focus on may be out of focus. In the above-mentioned process of focusing based on saliency detection, the original image must first be processed by the image front-end processing layer and the image processing engine to obtain an RGB image, and then the saliency detection network of the image processing layer performs saliency detection based on the RGB image, which obviously has low detection efficiency. In addition, in the process of the image front-end processing layer and the image processing engine processing the original image to obtain an RGB image, the image data of the final RGB image may be biased or missing, that is, the processed RGB image may not accurately express the currently captured scene, and thus there will be a certain deviation in the subsequent saliency detection.

[0043] In view of this, the embodiment of the present application provides a focusing method and related equipment, wherein the electronic device obtains the original image output by the image sensor, and obtains the depth image corresponding to the original image output by the depth sensor, and the electronic device performs saliency detection according to the original image and the depth image corresponding to the original image, obtains the saliency area, and then focuses based on the saliency area. The original image is used to detect the saliency area during the focusing process of the electronic device, which reduces the process of processing the original image (such as converting to RGB format) and improves the detection efficiency. Further, the error existing in the processing of the original image is avoided, and the data output by the image sensor and the depth sensor can accurately reflect the information of the current scene being photographed, thereby performing saliency detection based on the data output by the image sensor and the depth sensor, which can improve the robustness and accuracy of the saliency area detection result. Further, the feature information of the original image is combined with the feature information in the depth image corresponding to the original image, thereby further improving the accuracy of the image saliency detection result, making the saliency area in the image more accurately appear, providing accurate and useful information for subsequent electronic device focusing and other applications, and improving the accuracy of the determined focus frame.

[0044] The focusing method provided in the embodiment of the present application can be performed by an electronic device. The electronic device can be a terminal device, which can also be referred to as a terminal (terminal), user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device can be a smart phone, a computer, a smart TV, a personal digital assistant (PDA), a wearable device, an augmented reality (AR)\virtual reality (VR) device, a media player and other portable mobile devices. The electronic device can also be a vehicle-mounted device, an Internet of Things device or other device capable of shooting. The embodiment of the present application does not specifically limit the type of electronic device.

[0045] See also Figure 3 , the hardware structure of the electronic device 300 provided in the embodiment of the present application is exemplarily introduced. For the convenience of description, only various aspects of the image processing technology related to the embodiment of the present application are shown.

[0046] The electronic device 300 includes a central processing unit 301 , an imaging device 302 , a sensor module 303 , and an image processor 304 .

[0047] The CPU 301 is used to run an application 311 and an operating system (not shown). The application 311 may be a camera application, a gallery application, an instant messaging application, etc. The operating system may be an Android system, an IOS system, a Windows system, etc. The embodiment of the present application does not specifically limit the application 311 and the operating system run by the electronic device 300.

[0048] The imaging device 302 is used to shoot the current scene and capture image data. The imaging device 302 can be implemented as a camera in the electronic device 300, such as at least one camera. The at least one camera includes but is not limited to the following cameras: a rear camera, a front camera, and a rotating camera.

[0049] In the embodiment of the present application, the imaging device 302 at least includes a lens 312 (lens), an image sensor 322 (sensor), and a focus motor 332 .

[0050] The lens 312 is used to focus the optical information of the shooting scene on the image sensor 322 through the principle of optical imaging. The lens 312 and the focus motor 332 can be connected by an adhesive medium. When focusing is required, the electronic device 300 drives the focus motor 332 to push the lens 312, so that the distance (focal length) between the lens 312 and the image sensor 322 and the distance (object distance) between the lens 312 and the object to be photographed changes. The lens 312 includes but is not limited to: a wide-angle lens, an ultra-wide-angle lens, a telephoto lens and / or a periscope lens, etc.

[0051] The focus motor 332 can be implemented as a voice coil motor (VCM). Exemplarily, the focus motor 332 includes but is not limited to a coil, a magnet group and a spring. The coil is fixed in the magnet group by two upper and lower springs. When the coil is energized, the coil generates a magnetic field, and the coil magnetic field interacts with the magnet group, the coil moves upward, and the lens 312 locked in the coil moves together. When the power is off, the coil returns under the elastic force of the spring, and the lens 312 locked in the coil returns together accordingly, thereby realizing the movement of the lens 312 by the focus motor 332.

[0052] Among them, the image sensor 322 is a device that captures a two-dimensional or three-dimensional image of the object being photographed. The image sensor 322 can be a semiconductor chip. Hundreds of thousands to millions of photodiodes (such as single photon avalanche diodes (SPAD)) are arranged on the surface of the image sensor 322. When the photodiode is irradiated with light, it will generate an electric charge, thereby converting the light signal into an electrical signal. The image sensor 322 can be, for example, a charge coupled device (CCD), a complementary metal-oxide semiconductor device (CMOS), etc.

[0053] In the embodiment of the present application, the image sensor 322 may include a sensor array (not shown) and one or more analog-to-digital converters (not shown). The sensor array may be a CMOS sensor array or a CCD sensor array, and its resolution may be 1.3 million pixels, 3 million pixels or higher. The analog-to-digital converter may provide 6-bits, 8-bits or 10-bits accuracy according to application requirements, and convert the analog signal representing the image of the shooting scene generated by the sensor array into digital image data according to this accuracy. Here, the analog signal representing the image of the shooting scene generated by the sensor array may also be called an analog signal image.

[0054] In the embodiment of the present application, the image sensor 322 generates a raw image of the object through a sensor array and one or more analog-to-digital converters, and the generated raw image records the raw information of the sensor array. The raw image is in RAW format, and the RAW format indicates that it has not been processed in any way. The raw image in RAW format records the information of the sensor array, such as the data contained in the raw image can be the raw data of the CMOS sensor array or the CCD sensor array converting the captured light signal into a digital signal.

[0055] In the embodiment of the present application, the original image generated by the image sensor 322 may refer to an image generated by an analog image signal generated by a sensor array, an image generated by digital image data converted by an analog-to-digital converter, or an image generated by an image signal output by a phase pixel of the image sensor 322. The original image generated by the image signal output by the phase pixel of the image sensor 322 is a raw image composed of phase detection (PD) points, and the raw image composed of PD points includes a left phase image and a right phase image output by the image sensor 322 of the electronic device 300. The left phase image is a left pixel PD raw image output by the image sensor 322 of the electronic device 300, and the right phase image is a right pixel PD raw image output by the image sensor 322 of the electronic device 300.

[0056] Among them, the sensor module 303 includes at least a depth sensor. The depth sensor can be any sensor that can directly or indirectly obtain a depth image (depth image). Exemplarily, the depth sensor can be a sensor such as a millimeter wave radar or a laser radar. In other embodiments, the depth sensor can also be any sensor that can obtain a depth image and a grayscale image corresponding to the depth image. For example, the depth sensor can include a binocular camera, a monocular camera, a TOF camera, and other sensors.

[0057] In an embodiment of the present application, a depth image may also be referred to as a distance image, and each pixel in the depth image represents the distance between an object at a specific coordinate in the field of view of the depth sensor and the depth sensor. In other words, a depth image refers to an image that uses the distance (depth) from the depth sensor to each point in the shooting scene as a pixel value. The pixel value (or grayscale value) of each pixel in the depth image can represent a relative depth or an absolute depth (ie, a depth value). Relative depth can be understood as the logical relationship between each pixel. Among them, the distance between the real object corresponding to the pixel and the depth sensor or the reference plane is called the depth value of the pixel, that is, the depth value of the pixel is the distance from the real object corresponding to the point to the depth sensor.

[0058] In the embodiment of the present application, a depth image is captured by a depth sensor and transmitted to the image processor 304 , which can save the step of performing depth estimation based on image information, thereby saving the computing power of the image processor 304 .

[0059] The image processor 304 is used to obtain the raw image output by the image sensor 322 and process the raw image to generate image and / or video content. The image processor 304 is also used to analyze the raw image to capture status information that can be used to determine and / or control one or more control parameters of the imaging device 302.

[0060] In an embodiment of the present application, the image processor 304 includes an image front end 305 (image front end, IFE). The image front end 305 obtains the original image through an interface connected to the image sensor 322, and obtains the depth image through an interface connected to the depth sensor. The image front end 305 is used to pre-process the received data such as the original image and / or the depth image, such as time domain filtering. The image front end 305 is also used to collect the status information of the original image. The status information includes but is not limited to: automatic white balance status information, automatic exposure status information and automatic focus status information.

[0061] In the embodiment of the present application, the image front end 305 is also used to perform image registration on the obtained original image and the depth image, and input the original image after image registration and the depth image corresponding to the original image to the focusing device 307. The focusing device 307 performs the focusing method provided in the embodiment of the present application on the original image and the depth image corresponding to the original image.

[0062] In an embodiment of the present application, image registration is the process of converting different images of the same scene into the same coordinate system. These different images may be taken at different times (multi-time registration), may be taken by different sensors (multi-mode registration), and may be taken from different perspectives. The spatial relationship between these images may be rigid (translation and rotation), affine (such as miscut), or may be homographic, or a complex large-scale deformation model. The image front end 305 performs image registration on the obtained original image and the depth image, that is, converts the original image and the depth image to the same coordinate system.

[0063] In the embodiment of the present application, the image front end 305 has a built-in focusing device 307. The focusing device 307 includes a saliency detection module 327, which is used to perform saliency detection on the original image and the depth image to extract the most attention-grabbing object (i.e., the salient subject), and then determine the area where the salient subject is located in the original image (i.e., the salient area). The saliency detection module 327 can be a pre-trained neural network model, and the electronic device 300 can obtain the salient area in the original image based on the pre-trained neural network model.

[0064] Preferably, the saliency detection module 327 is used to perform saliency detection on the original image and the depth image that have undergone image registration.

[0065] In some other embodiments, the focusing device 307 may further include a preprocessing module 317. The preprocessing module 317 is used to determine whether the picture of the scene captured by the electronic device 300 belongs to a multi-depth scene. When the picture of the current scene captured by the electronic device 300 belongs to a multi-depth scene, the foreground and background of the original image are segmented to obtain a processed original image. The background of the processed original image is covered relative to the original image before processing.

[0066] When the preprocessing module 317 performs foreground and background segmentation on the original image, the preprocessing module 317 inputs the depth image and the processed original image to the saliency detection module 327. The saliency detection module 327 performs saliency detection on the depth image and the processed original image, and outputs a salient region.

[0067] In other embodiments, the preprocessing module 317 can be used to obtain the original image and the depth image from the image front end 305, and perform image registration on the obtained original image and depth image, and input the original image (or processed original image) and the depth image after image registration into the saliency detection module 327.

[0068] In some other embodiments, the focusing device 307 may further include a frame-out module 337. The frame-out module 337 is used to process the salient regions output by the salient detection module 327 to determine a target focus frame. When the number of salient regions output by the salient detection module 327 is 1, the frame-out module 337 determines the target focus frame based on the salient regions. When the number of salient regions output by the salient detection module 327 is two or more, the frame-out module 337 may select the salient region with the highest confidence from the two or more salient regions as the target region, and then determine the target focus frame based on the target region.

[0069] The preprocessing module 317, the saliency detection module 327 and the frame-out module 337 in the focusing device 307 may be a program running on the electronic device 300 to implement corresponding functions. In subsequent examples, the functions of the above modules will be described in detail.

[0070] In the embodiment of the present application, the image processor 304 may further include an autofocus module 306. The autofocus module 306 is connected to the image front end 305 via a first interface, and the autofocus module 306 may obtain the target focus frame output by the image front end 305 via the first interface. The autofocus module 306 drives the focus motor 332 to move the lens 312 based on the target focus frame.

[0071] In other embodiments, the focusing device 307 provided in the embodiment of the present application may be built into the auto-focus module 306. The auto-focus module 306 and / or the focusing device 307 may obtain the original image and the depth image corresponding to the original image from the image front end 305 through the first interface. The focusing device 307 obtains the original image and the depth image corresponding to the original image from the first interface, and executes the focusing method provided in the embodiment of the present application.

[0072] In the embodiment of the present application, the image processor 304 may include one or more processors, such as one or more image signal processors (ISP), one or more host processors and / or one or more other types of processors in any other type of processor discussed with respect to the computing system. The host processor may be a digital signal processor (DSP) and / or other types of processors.

[0073] In the embodiment of the present application, the application 311 issues a first instruction in response to the user's operation, and the first instruction instructs the electronic device 300 to shoot. The central processor 301 issues the first instruction to the imaging device 302 and the depth sensor to start the depth sensor and the image sensor 322 in the imaging device 302. After the imaging device 302 is started, the object in the current scene obtains a Bayer image through the lens 312. After the Bayer image is input to the image sensor 322, an analog electrical signal is obtained by photoelectric conversion, and a digital image signal is further obtained by processing the analog-to-digital converter, thereby obtaining an original image. After the depth sensor is started, a depth image of the object in the current scene is obtained. The image front end 305 in the image signal processor obtains the original image output by the image sensor 322 and the depth image output by the depth sensor. The focusing device 307 in the image signal processor performs saliency detection on the original image and the depth image corresponding to the original image to obtain a saliency area. The focusing device 307 also outputs a target focus frame to the autofocus module 306 according to the saliency area. The auto focus module 306 drives the focus motor 332 based on the target focus frame, and then drives the lens 312 through the focus motor 332 to focus on the object in the target focus frame.

[0074] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a microservice architecture or a cloud architecture. The following embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device. Of course, in other operating systems, as long as the functions implemented by each functional module are similar to those of the embodiment of the present application, the embodiment of the present application can also be implemented.

[0075] See also Figure 4 , which exemplarily introduces the software structure of the electronic device provided in the embodiments of the present application.

[0076] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, application layer 410, hardware abstraction layer 420, driver layer 430, and hardware layer 440.

[0077] The application layer 410 may include a series of application packages, which may include a camera application, a gallery, and an application that can provide a shooting function, such as an instant messaging application.

[0078] The hardware abstraction layer 420 is used to abstract the hardware. For example, the hardware abstraction layer 420 may include a camera abstraction layer and other hardware device abstraction layers. The camera abstraction layer may include imaging device 1, imaging device 2, etc. The camera hardware abstraction layer 420 may be connected to an image processing algorithm library, and the camera hardware abstraction layer 420 may call algorithms in the image processing algorithm library. For example, the image processing algorithm library may include algorithm instructions such as camera algorithms, image algorithms, 3A algorithms, and saliency detection algorithms.

[0079] The driver layer 430 is used to provide drivers for different hardware devices. For example, the driver layer 430 may include a camera device driver.

[0080] The hardware layer 440 may include a depth sensor, an image processor, an image sensor, a focus motor, a lens, and other hardware devices.

[0081] In the embodiment of the present application, the user issues a first instruction through an application program, and the first instruction can be transmitted to the depth sensor and the light-sensitive sensor of the hardware layer 440 through the hardware abstraction layer 420 and the driver layer 430. In some implementations, the user clicks the icon of the camera application on the desktop of the electronic device, and the camera application detects the shooting operation and issues the first instruction to the camera hardware abstraction layer 420. The camera hardware abstraction layer 420 sends an image sensor start instruction to the camera device driver to control the image sensor to start. At the same time, the camera hardware abstraction layer 420 also sends a depth sensor start instruction to the camera device driver to control the depth sensor to start.

[0082] The image processor obtains the depth image output by the depth sensor and the original image output by the image sensor. The image signal processor calls image processing related algorithms through the camera device driver and the image processing algorithm library of the camera hardware abstraction layer 420. For example, the saliency detection algorithm is called to perform saliency detection on the original image and the depth image, and after determining the target focus frame, the focus motor is driven based on the autofocus algorithm in the 3A algorithm, and the focus motor moves the lens.

[0083] See also Figure 5 , Figure 5 is a flow chart of a focusing method provided in an embodiment of the present application, and the focusing method is applicable to the above Figure 3 Any one of the focusing devices, electronic devices, and electronic devices including the focusing device is also applicable to Figure 4 The electronic device shown in the figure includes an image sensor and a depth sensor, and the focusing method may include the following steps.

[0084] Step S51: the electronic device starts the image sensor and the depth sensor in response to the first instruction.

[0085] In an embodiment of the present application, an application with a shooting function is installed in the electronic device, such as a camera application or an instant messaging application. When a user wants to take a picture, shoot a video or make a video call, the user operates the corresponding application with a shooting function. The application sends a first instruction in response to the user's shooting operation, and the first instruction is used to instruct the electronic device to shoot the current scene. In response to the first instruction, the electronic device starts the image sensor and depth sensor of the electronic device.

[0086] The relevant contents of the image sensor and the depth sensor can be referred to above and will not be repeated here.

[0087] In step S52, the electronic device obtains an original image output by the image sensor, and obtains a depth image corresponding to the original image output by the depth sensor.

[0088] In the embodiment of the present application, in step S51, the electronic device starts the image sensor and the depth sensor. After the image sensor is started, the image sensor collects image data in the current scene to obtain an original image. After the depth sensor is started, the depth sensor collects the distance of the object in the current scene to obtain a depth image. Among them, the relevant contents of the original image and the depth image can be referred to above, and will not be repeated here.

[0089] In the embodiment of the present application, the electronic device may obtain the depth image output by the depth sensor and corresponding to the original image by, for example, obtaining the original image output by the image sensor, obtaining the depth image output by the depth sensor, and performing image registration on the obtained original image and the depth image. That is, the depth image corresponding to the original image may be the depth image after performing image registration on the original image and the depth image.

[0090] It can be understood that the original image and the depth image can be acquired synchronously or asynchronously, and the embodiments of the present application do not specifically limit this.

[0091] In step S53, the electronic device inputs the original image and the depth image to a focusing device, and processes the original image and the depth image based on the focusing device to obtain a salient area on the original image.

[0092] In the embodiment of the present application, the electronic device performs saliency detection on the original image and the depth image based on the saliency detection module in the focusing device to obtain a saliency region. The saliency region is the image region where the salient subject is located. When the saliency detection module uses a person as the salient subject, Figure 1B The image area where the character portrait is located is the salient area.

[0093] The saliency detection module can be implemented as a neural network. Through model training, the neural network can learn which objects in the image are more salient. For example, it is generally believed that people and cars are more salient than walls, sky, grass, and distant buildings, so people and cars are salient subjects. In addition, other methods (such as key area focusing methods, etc.) can also be used to identify salient areas, which are not limited here. Among them, using a machine learning model such as a neural network to identify salient areas can make the identification more intelligent compared to other methods of identifying salient areas.

[0094] In some embodiments, multiple models may be trained based on different saliency definitions for user selection, such as a "face priority" saliency model, a "text priority" saliency model, and the like.

[0095] In an embodiment of the present application, the salient detection module includes an encoder and a decoder. The electronic device inputs the original image and the depth image into the trained encoder, and based on the parameters of the encoder, performs feature extraction on the original image and the depth image to obtain a first feature image and a second feature image, respectively. The electronic device inputs the first feature image and the second feature image into the trained decoder, and based on the parameters of the decoder, fuses the first feature image and the second feature image to obtain a salient area on the original image.

[0096] Exemplarily, taking the above-mentioned trained encoder including the first encoder and the second encoder as an example, the original image is input into the first encoder, and the feature extraction of the original image is performed based on the parameters of the first encoder to obtain the first feature image. The depth image is input into the second encoder, and the feature extraction of the depth image is performed based on the parameters of the second encoder to obtain the second feature image. Exemplarily, the first encoder can be a convolution-deconvolution model, wherein the convolution stage is used as a feature extractor to convert the original image into a hierarchical and rich feature representation, and the deconvolution stage can be used as a shape restorer to restore the information in the original image, such as texture, resolution, etc., and to subdivide the salient objects in the details from the background. The first encoder can also be a convolutional neural network (CNN), an encoder based on a transformer (Transformer) structure, etc. Exemplarily, the second encoder can be a convolutional neural network including multiple layers, wherein each subsequent layer reduces the dimension of the extracted features to extract depth features.

[0097] After the electronic device inputs the original image into the first encoder, it can obtain the feature layer of the encoded original image, that is, the first feature image. After the electronic device inputs the depth image into the second encoder, it can obtain the depth information feature layer of the depth image, that is, the second feature image. The electronic device inputs the obtained feature layer of the original image and the depth information feature layer of the depth image into the trained decoder, and fuses the complementary semantic information between the first feature image and the second feature image based on the parameters of the decoder to generate a joint feature image, that is, a saliency image (RAW-D saliency map). The saliency image can indicate the salient area in the original image, and the salient area can be obtained according to the saliency image.

[0098] The parameters of the above encoders and decoders are all parameters that need to be obtained through a lot of training in practical applications, including but not limited to bias parameters and weight parameters. Taking the convolutional neural network as an example, each weight matrix formed by the weight parameters obtained through training plus the bias parameters obtained through training can extract information from the input image data, thereby helping the convolutional neural network to perform effective feature extraction.

[0099] In other embodiments, the saliency detection module can be implemented as a convolutional neural network including two input layers, one or more hidden layers and an output. The original image and its corresponding depth image are respectively used as the original input images of the two input layers, and are input into the convolutional neural network for training to obtain a trained convolutional neural network. The original image obtained by the image front end and the depth image corresponding to the original image are respectively used as the original input images of the two input layers, and are input into the trained convolutional neural network for training, and the trained parameters such as the optimal weight vector and the optimal bias item are used for prediction to obtain the saliency image corresponding to the original image, thereby realizing saliency subject detection.

[0100] It can be understood that the structure of the above-mentioned saliency detection module is only an example, and the saliency detection module can also be set according to actual conditions. The above-mentioned encoder and decoder are only examples, and an encoder for extracting feature information of the original image (such as a pd raw image) and an encoder for extracting feature information of the depth image can be set according to actual conditions. The decoder can also be set according to actual conditions to fuse the extracted feature information of the original image and the feature information of the depth image. The embodiment of the present application does not specifically limit the specific structural types of the saliency detection module, encoder, and decoder.

[0101] In an embodiment of the present application, when acquiring the salient area of ​​the original image, the saliency detection module combines the feature information of the original image (such as texture) and the feature information of the depth image (such as depth), and effectively integrates the feature information of the depth image with the feature information of the original image. Therefore, the saliency detection of the original image can be more accurate, thereby improving the robustness of the detection results.

[0102] In the embodiment of the present application, the electronic device determines whether the picture of the current scene belongs to a multi-depth scene through a focusing device, and when the picture of the scene is determined to be a multi-depth scene, the foreground and background of the original image are segmented to obtain a processed original image. Then, the processed original image and the depth image are input into a saliency detection module, such as an encoder trained in the saliency detection module.

[0103] In the embodiment of the present application, the image may include a foreground and a background. The foreground is the area where the subject of the image is located, generally the area where the shooting focus is located, and the background is the area other than the foreground in the image. Figure 1B The person portrait in the preview image shown is the background, and the hand and the pen are the foreground.

[0104] In an embodiment of the present application, when the electronic device determines that the picture of the current scene is a multi-depth scene, the electronic device determines the depth statistics of the areas where each subject in the original image is located based on the depth image corresponding to the original image. The maximum depth statistic in the depth image is used as the segmentation threshold. The electronic device processes the original image according to the segmentation threshold to obtain a processed original image, wherein the depth statistics of the uncovered area in the processed original image is less than the segmentation threshold, and the depth statistics of the covered area in the processed original image is greater than or equal to the segmentation threshold. For each area, the depth statistic can be the median of the depth values ​​corresponding to each pixel in the area, or the average value obtained by averaging the sum of the depth values ​​corresponding to all pixels in the area, or the sum of the depth values ​​corresponding to all pixels in the area. The present application does not specifically limit this depth statistic.

[0105] In some embodiments, the electronic device can input the original image into the foreground-background segmentation model to obtain the processed original image. The foreground-background segmentation model is a function that can obtain an output from an input, the input is an image, and the output is a segmentation result. The foreground-background segmentation model can be used to identify the foreground and background in an image and separate the foreground and background of the image. The segmentation result refers to the result obtained after dividing the image into two areas, the foreground and the background, and the processed original image is obtained based on the segmentation result.

[0106] In the embodiment of the present application, the electronic device determines whether the current scene belongs to a multi-depth scene in the following ways, but is not limited to:

[0107] Method 1: The electronic device determines the number of depth regions in the depth image, and each depth region corresponds to a unique depth value range. When the number of depth regions in the depth image is greater than or equal to a preset threshold, it is determined that the picture of the current scene belongs to a multi-depth scene; when the number of depth regions in the depth image is less than the preset threshold, it is determined that the picture of the current scene belongs to a multi-depth scene.

[0108] The depth region indicates a continuous image region in the depth image, and the depths (such as relative depths or depth values) of the pixels in the image region are all within the same depth value range. The preset threshold may be a quantitative threshold for indicating the depth region, which may be set according to actual conditions, such as 2, 3, 4, 5, etc., and the present application embodiment does not specifically limit this.

[0109] For example, taking the depth value range including the first depth value range to the third depth value range as an example, the first depth value range is greater than or equal to 50 cm and less than 1 meter, the second depth value range is greater than or equal to 1 meter and less than 1.5 meters, and the third depth value range is greater than 1.5 meters and less than or equal to 2 meters. Figure 1B In the preview image shown, the depths corresponding to all the pixel points in the image area where the hand and the pen are located are within the first depth value range, so the image area where the hand and the pen are located is a depth area. Figure 1B If the depths corresponding to all pixels in the image area where the person portrait in the preview image is located are within the third depth value range, then the image area where the person portrait is located is another depth area. Figure 1B The picture belongs to a multi-depth scene.

[0110] Method 2: The electronic device obtains a phase difference sequence composed of multiple PD values. Then, based on the phase difference sequence and the difference threshold, it is determined whether the framing picture belongs to a multi-depth scene. The electronic device determines the maximum and minimum values ​​in the phase difference sequence, and determines the absolute value of the difference between the maximum and minimum values ​​and the preset threshold. If the absolute value of the difference is greater than the difference threshold, the scene is determined to be a multi-depth scene. If the absolute value of the difference is less than the difference threshold, the scene is determined to be a non-multi-depth scene. The difference threshold is a threshold for indicating the absolute value of the difference in PD values, which can be set according to actual conditions, such as the difference threshold is 80.

[0111] Exemplarily, the electronic device imaging device is configured with PD pixels and has a PDAF function (supporting PDAF technology). That is, the image taken by the imaging device can obtain a phase difference sequence through the processing of the PDAF algorithm. Alternatively, the electronic device inputs the PD raw image into the phase detection library to calculate the PD value. Among them, the phase detection library can increase the gain (gain) for the PD pixel corresponding to the PD raw image through a gain map. Then, the PD value is calculated based on the PD pixel after the gain is increased, and the phase difference sequence is obtained based on the multiple PD values.

[0112] Taking the phase difference sequence {pd1, pd2, pd3, pd4, ..., pdn} and the difference value of 80 as an example, the maximum pd value max_pd and the minimum pd value min_pd are selected in the phase difference sequence {pd1, pd2, pd3, pd4, ..., pdn}; when |max_pd-min_pd| is greater than or equal to 80, the scene is judged to be a multi-depth scene. When max_pd-min_pd| is less than 80, the scene is judged to be a non-multi-depth scene.

[0113] In the embodiment of the present application, a mask (mask image) can be used to block part or all of the original image. In the embodiment of the present application, the mask image is used to block the background in the original image. Regarding the actual form of the mask image, the present application does not specifically limit this.

[0114] In step S54, the electronic device focuses on the object in the current scene according to the saliency area.

[0115] In the embodiment of the present application, the frame output module in the focusing device is used to determine the target focus frame based on the salient area output by the salient detection module, so that the autofocus module focuses based on the target focus frame, that is, the target focus frame is the focus area of ​​the lens focus. The target focus frame frames the photographed object (i.e., the salient subject) in the original image, and instructs the electronic device to use the area framed by the target focus frame (the salient subject) as the focus area.

[0116] In the embodiment of the present application, the frame-out module obtains the minimum bounding rectangle area of ​​the salient area, such as obtaining the minimum bounding rectangle of the salient area according to the convex hull method, and the maximum area included in the minimum bounding rectangle is the minimum bounding rectangle area. Then, the minimum bounding rectangle area corresponding to the salient area is determined as the focus frame corresponding to the salient area.

[0117] It can be understood that the rectangle in the minimum circumscribed rectangular area corresponding to the above-mentioned salient area is only an example, and can also be an ellipse or a circle, etc., which is not specifically limited here. That is, when the electronic device can display a focus frame, the focus frame can be a rectangle such as Figure 1B The rectangle shown may also be an ellipse or a circle.

[0118] In the embodiment of the present application, the salient area is the area where the salient subject is located, and the electronic device can use an auto focus (AF) algorithm to focus the lens on the salient subject based on the area where the salient subject is located, that is, use the AF algorithm to focus the lens on the salient subject framed by the target focus frame.

[0119] In other embodiments, the electronic device may pre-store the correspondence between the depth range and the focus position corresponding to the depth range, and then determine the target focus position of the focus motor according to the depth of the area where the significant subject is located (such as depth statistics), and then drive the focus motor to the target focus position, thereby focusing the lens on the significant subject.

[0120] Specifically, the electronic device may store a comparison table according to the correspondence between the depth range and the focus position corresponding to the depth range, and the comparison table records the depth range and the focus position corresponding to the depth range. In some embodiments, the correspondence between the depth range and the focus position stored in the memory of the electronic device may be as shown in Table 1 below, where the depth range in Table 1 may be the distance range between the center of the target object and the lens, in centimetres (cm), and the focus position is the focus position to which the focus motor drives the lens to move.

[0121] Table 1

[0122] Depth range (cm) Focus position Greater than 10 or equal to and less than or equal to 75 50 Greater than 75 and less than 125 100

[0123] When the depth statistic of the target area is 50, it belongs to the depth range greater than 10 or equal to and less than or equal to 75, then the target focus position is determined to be 50, and the electronic device drives the focus motor to move the lens to the position indicated by the focus position 50.

[0124] It can be understood that the depth range and focus position settings in Table 1 above can be set according to actual conditions, and this application does not make any specific limitations on this.

[0125] In the embodiments of the present application, the "target focus position" may include a moving direction and / or a moving distance. The moving direction indicates the direction in which a movable lens, lens group or lens in a camera needs to move in order to obtain a clear image; the moving distance indicates the distance in which a movable lens, lens group or lens in a camera needs to move in the moving direction in order to obtain a clear image. In other words, the focus position may indicate the position in which a movable lens, lens group or lens in a camera needs to be in order to obtain a clear image.

[0126] In the embodiment of the present application, when the number of salient regions output by the saliency detection module is two or more, please refer to Figure 6 Step S54 may specifically include:

[0127] In step S61, the electronic device obtains, for each salient region, a saliency statistic, a depth statistic, and a first distance corresponding to the saliency region, wherein the first distance is used to indicate a distance between the saliency region and a central region of an original image.

[0128] The significance statistic may be the median of the significance values ​​of all pixels in the significance region, the average of the significance values ​​of all pixels in the significance region, or the sum of the significance values ​​of all pixels in the significance region.

[0129] The depth statistics can be referred to above and will not be described in detail here.

[0130] The first distance may be the distance between the center point of the salient area and the center point of the center area of ​​the original image, or may be the distance between a first edge point of the salient area and a second edge point of the center area of ​​the original image, wherein the first edge point is the edge point on the salient area that is closest to the center area, and the second edge point is the edge point on the center area that is closest to the salient area.

[0131] In the embodiment of the present application, the saliency detection is performed on the image, and the saliency value of each pixel of the image can be obtained. In other words, the saliency image includes the saliency value of each pixel in the image. Accordingly, for each saliency region, the saliency value of all pixels in the saliency region can be obtained, the sum of the saliency values ​​of all pixels in the saliency region is calculated, and the sum is averaged to obtain the average value of the saliency values ​​of all pixels in the saliency region.

[0132] In step S62, the electronic device calculates the confidence level of the salient region according to the saliency statistics, the depth statistics and the first distance.

[0133] In the embodiment of the present application, corresponding weights can be set for the significance statistics, the depth statistics and the first distance according to actual conditions, and then the significance statistics, the depth statistics and the first distance are comprehensively analyzed to determine the confidence level.

[0134] Exemplarily, for each salient region, the electronic device may calculate the confidence of the salient region according to the following first formula:

[0135] Y=k1*a+k2*b+k3*c

[0136] Among them, k1 is the weight set by the electronic device for the significance statistic, k2 is the weight set by the electronic device for the depth statistic, k3 is the weight set by the electronic device for the first distance, a is the significance statistic of the significance area, b is the depth statistic of the significance area, c is the first distance of the significance area, and Y is the confidence of the significance area.

[0137] It can be understood that the electronic device can set the weights of k1 to k3 according to actual conditions, and this application does not make any specific limitation on this.

[0138] In step S63, the electronic device determines the salient area with the highest confidence as the target area.

[0139] In step S62, for each salient region, the electronic device obtains the confidence of the salient region, compares the confidences of the salient regions, and determines the salient region with the highest confidence as the target region.

[0140] Step S64: the electronic device focuses on the object in the current scene according to the target area.

[0141] In an embodiment of the present application, the electronic device determines a target focus position according to a target area, and the electronic device drives a focus motor to move the lens to the target focus position so that the lens focuses on a prominent subject within the target focus position in the current scene.

[0142] The present application also provides an electronic device, which includes Figure 3 , Figure 4 In some other embodiments, the electronic device includes at least one or more processors and one or more memories. The processor includes a central processing unit and a graphics processing unit. The one or more memories are coupled to the one or more processors, and the one or more memories store computer instructions; when the one or more processors execute the computer instructions, the electronic device executes the above Figure 5 , Figure 6 Focusing method shown.

[0143] See also Figure 7 , Figure 7 A schematic diagram of a chip system structure provided in an embodiment of the present application.

[0144] Figure 7A schematic diagram of the composition of a chip system 700 is shown. The chip system 700 may include: a processor 701 and a communication interface 702, which are used to support related devices to implement the functions involved in the above embodiments. In a possible design, the chip system also includes a memory for storing necessary program instructions and data for electronic devices. The chip system may be composed of chips, or may include chips and other discrete devices. It should be noted that in some implementations of the present application, the communication interface 702 may also be referred to as an interface circuit.

[0145] Specifically, the chip system is applied to an electronic device, which includes an image sensor and a depth sensor, and is characterized in that the chip system includes an interface circuit and a processor; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send a signal to the processor, and the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the chip system responds to a first instruction to start the image sensor and the depth sensor, wherein the first instruction is used to instruct to shoot the current scene; obtain an original image output by the image sensor, and obtain a depth image corresponding to the original image output by the depth sensor; input the original image and the depth image into a trained encoder, and based on the parameters of the encoder, extract features from the original image and the depth image to obtain a first feature image and a second feature image respectively; input the first feature image and the second feature image into a trained decoder, and based on the parameters of the decoder, fuse the first feature image and the second feature image to obtain a salient area on the original image; focus on objects in the current scene according to the salient area.

[0146] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0147] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the focusing method in the above-mentioned method embodiments.

[0148] An embodiment of the present application further provides a computer storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the focusing method as described in the above embodiment.

[0149] Among them, the electronic device, computer storage medium, computer program product or chip system provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0150] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0151] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0152] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0153] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0154] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0155] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A focusing method, characterized in that: Applied to an electronic device, the electronic device includes an image sensor and a depth sensor, and the method includes: In response to a first instruction, starting the image sensor and the depth sensor, wherein the first instruction is used to instruct to shoot a current scene, wherein the current scene is a multi-depth scene; Acquire a RAW image output by the image sensor, and acquire a depth image output by the depth sensor corresponding to the RAW image; Performing image registration on the RAW image and the depth image; Performing foreground and background segmentation on the RAW image after image registration to obtain a processed RAW image, wherein the background in the processed RAW image is blocked; Inputting the processed RAW image and the depth image after image registration into a trained encoder, and performing feature extraction on the processed RAW image and the depth image after image registration based on parameters of the encoder to obtain a first feature image and a second feature image respectively; Inputting the first feature image and the second feature image into a trained decoder, and fusing the first feature image and the second feature image based on parameters of the decoder to obtain a salient area on the processed RAW image; When the number of the salient regions is two or more, for each of the salient regions, obtaining a salient statistic value, a depth statistic value, and a first distance corresponding to the salient region, wherein the first distance is used to indicate a distance between the salient region and a central region of the processed RAW image; Calculating the confidence of the salient region according to the saliency statistics, the depth statistics, and the first distance; Determine the significant region with the highest confidence as the target region; Focusing on objects in the current scene according to the target area.

2. The method according to claim 1, characterized in that: The focusing on the object in the current scene according to the target area comprises: Determine a target focus position according to the target area; The focus motor is driven to move the lens to the target focus position to focus on the object in the current scene.

3. The method according to claim 2, characterized in that: Determining the target focus position according to the target area includes: Obtaining a comparison table pre-stored in the electronic device, wherein the comparison table records a depth range and a focus position corresponding to the depth range; The target focus position is determined according to the depth statistics of the target area and the comparison table.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire the number of depth regions in the depth image, each of the depth regions corresponding to a unique depth range; When the number of depth regions in the depth image is greater than or equal to a preset threshold, determining that the picture of the current scene belongs to a multi-depth scene; When the number of depth regions in the depth image is less than the preset threshold, it is determined that the picture of the current scene does not belong to a multi-depth scene.

5. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, wherein the computer program includes program instructions; The processor is used to call the computer program to execute the method according to any one of claims 1 to 4.

6. A chip system, the chip system is applied to an electronic device, the electronic device includes an image sensor and a depth sensor, characterized in that: The chip system includes an interface circuit and a processor; the interface circuit and the processor are interconnected through a line; the interface circuit is used to receive a signal from the memory of the electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, The chip system starts the image sensor and the depth sensor in response to a first instruction, wherein the first instruction is used to instruct to shoot a current scene, wherein the current scene is a multi-depth scene; Acquire a RAW image output by the image sensor, and acquire a depth image output by the depth sensor corresponding to the RAW image; Performing image registration on the RAW image and the depth image; Performing foreground and background segmentation on the RAW image after image registration to obtain a processed RAW image, wherein the background in the processed RAW image is blocked; Input the processed RAW image and the depth image after image registration into a trained encoder, and based on the parameters of the encoder, perform feature extraction on the processed RAW image and the depth image after image registration to obtain a first feature image and a second feature image respectively; input the first feature image and the second feature image into a trained decoder, and based on the parameters of the decoder, fuse the first feature image and the second feature image to obtain a salient area on the processed RAW image; when the number of the salient areas is two or more, for each of the salient areas, obtain a salient statistic, a depth statistic, and a first distance corresponding to the salient area, wherein the first distance is used to indicate the distance between the salient area and the central area of ​​the processed RAW image; calculate the confidence of the salient area according to the salient statistic, the depth statistic, and the first distance; determine the salient area with the highest confidence as the target area; Focusing on objects in the current scene according to the target area.

7. A computer storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 4.

8. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 4.

Images