Image processing method and electronic equipment

Abstract

The embodiment of the invention provides an image processing method and electronic equipment, and is applied to the technical field of image processing. The electronic device includes a camera assembly. The method comprises the steps of controlling a camera assembly, and shooting a first area based on a first focal length to obtain a first image; and inputting a first image, and executing a first operation according to the first mapping relation, the first focusing motor value and the first focal length. The first mapping relation is the mapping relation between the focusing motor value of the camera assembly and the image distance. The first focus motor value is a theoretical focus motor value of the camera assembly with respect to a target image region in the first image. The first operation comprises at least one of the following: outputting a depth image about the target image area; outputting a neural network operation result about the target image region; and outputting a control instruction, wherein the control instruction is used for indicating the electronic equipment to execute a certain equipment operation on the target image area. According to the embodiment of the invention, the depth image is obtained on the basis of reducing the device cost and the data processing amount of the electronic equipment.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] With the development of electronic information automation technology, more and more intelligent electronic devices are integrated with camera components. The target area is photographed by the camera component to obtain a photographed image of the target area. According to the algorithm processing of the photographed image, a depth image of the target area can be obtained. The depth image includes the object distance information of the target area. According to the object distance information contained in the depth image, the intelligent electronic device can perform different algorithm processing to obtain the corresponding algorithm processing results. Perform various intelligent operations according to the algorithm processing results. In the development of modernization, more and more intelligent electronic devices have integrated the above-mentioned functions of obtaining depth images based on camera components and performing intelligent operations based on depth images.

[0003] The traditional method of acquiring depth images based on camera components is to integrate a ToF distance measuring instrument or a binocular camera component in an intelligent electronic device, and obtain a depth image of the target area based on the ToF distance measuring principle or the structured light distance measuring principle. However, this implementation method requires the additional installation of an expensive ToF distance measuring instrument or binocular camera, which makes the product cost high and requires more complex data processing to obtain a depth image. Summary of the invention

[0004] The embodiments of the present application provide an image processing method and an electronic device, which obtain a depth image while reducing device cost and data processing volume.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an image processing method is provided, which is based on an electronic device; the electronic device includes a camera component. The method includes: controlling the camera component to shoot a first area based on a first focal length to obtain a first image. Inputting the first image, performing a first operation according to a first mapping relationship, a first focus motor value, and a first focal length. The first mapping relationship is a mapping relationship between the focus motor value of the camera component and the image distance; the first focus motor value is a theoretical focus motor value of the camera component relative to a target image area in the first image. The first operation includes at least one of the following: outputting a depth image of the target image area; outputting a neural network operation result of the target image area; outputting a control instruction, the control instruction being used to instruct the electronic device to perform a certain device operation on the target image area.

[0007] In an embodiment of the present application, a first area is photographed based on a first focal length by a camera assembly to obtain a first image. A depth image is calculated based on the first focus motor value of the target image area and the image distance-focus motor value mapping relationship of the camera assembly (i.e., the first mapping relationship) and the first focal length, and the depth image is output. Alternatively, a neural network operation is performed based on the depth image to obtain a neural network operation result. Alternatively, a control instruction is output based on the depth image to control the electronic device to perform related intelligent operations. In the embodiment of the present application, there is no need to set up additional measuring instruments or more expensive camera components, and the depth image can be obtained with a universal device. At the same time, in the process of generating a depth image, a large amount of data calculation is not required, and a depth image can be generated based on known parameters and / or easily accessible parameters. Compared with the traditional solution that requires a large amount of calculation to generate a depth image, this solution can realize the calculation of the depth image in a cheaper and lighter electronic device, and various artificial intelligence solutions are applied based on the depth image. This solution greatly reduces the cost and data processing of various electronic devices based on depth images, so that such electronic equipment products can be generated, applied and popularized faster.

[0008] In a possible implementation, the first operation is performed according to the first mapping relationship, the first focus motor value and the first focal length, including: obtaining the target image distance according to the first mapping relationship and the first focus motor value; the target image distance is the distance between the lens of the camera assembly and the imaging of the target image area. Obtain the target object distance according to the target image distance and the first focal length; the target object distance is the distance between the target image area of ​​the first image and the camera assembly. Perform the first operation according to the target object distance. In the embodiment of the present application, each camera assembly will have a designed first mapping relationship before leaving the factory, that is, the image distance-focus motor value mapping relationship, which can be a linear relationship or a nonlinear relationship. Then, according to the first focus motor value of the target image area and the first mapping relationship, the target image distance of the target image area can be obtained. According to the target image distance and the first focal length, the target object distance of the target image area can be obtained. The depth image can be calculated according to the target object distance. In different application scenarios, different first operations are performed based on the depth image. Through the above operations, a depth image can be obtained with a small amount of data processing based on known or easily obtained parameters.

[0009] In a possible implementation, the method further includes: obtaining a first focus motor value according to the contrast of the target image area in the first image. Exemplarily, the first focus motor value can be calculated based on contrast detection autofocus (CDAF). In an embodiment of the present application, CDAF technology is a camera autofocus technology that determines the position of the focus by analyzing the contrast of the image. Compared with traditional phase focusing technology, CDAF technology has higher accuracy and a wider focus range. The advantage of CDAF technology is that it can achieve higher focus accuracy and a wider focus range, and can also achieve smooth focus transition in video shooting. After selecting the target image area in the first image, the target image area is used as the target focus area, and the first focus motor value of the quasi-focus can be determined by CDAF technology. The first focus motor value is calculated based on CDAF technology, which expands the application of CDAF technology. At the same time, CADF technology can be implemented based on ordinary camera sensing components without adding additional hardware, etc., which greatly reduces hardware costs.

[0010] In a possible implementation, the method further includes: obtaining a first focus motor value according to a phase difference of the target image area in the first image. Exemplarily, the first focus motor value can be calculated based on phase detection auto focus technology (PDAF). In an embodiment of the present application, the phase difference is calculated based on the PDAF technology, and the first focus motor value is obtained based on the phase difference, and the application of the PDAF technology is expanded based on this implementation. At the same time, this implementation only requires a single camera sensing module with phase detection function to be set in the camera assembly.

[0011] In a possible implementation, the above-mentioned obtaining the first focus motor value according to the phase difference of the target image area in the first image includes: obtaining a motor offset according to the phase difference of the target image area in the first image; the motor offset is the difference between the current focus motor value of the camera assembly and the first focus motor value. The first focus motor value is obtained according to the current focus motor value and the motor offset. In an embodiment of the present application, the phase difference of the target image area in the first image can be obtained based on the PDAF technology. The motor offset between the current focus motor value and the first focus motor value of the target can be calculated based on the phase difference. The first focus motor value can be obtained according to the motor offset and the current focus motor value.

[0012] For example, the motor offset may be calculated based on the phase difference and a defocus conversion coefficient (DCC) of the focus motor.

[0013] For example, the motor offset may be obtained based on the phase difference through artificial intelligence auto focus (AIAF) technology.

[0014] In a possible implementation, the above-mentioned certain device operation includes: adjusting the shooting parameters of the camera component. In the embodiment of the present application, in the application scenario where the camera component is required to shoot and image, the required shooting parameters can be calculated based on the depth image of the target image area, and the control instructions can be output. The shooting parameters of the camera component are adjusted according to the control instructions to achieve the purpose of better imaging of the target image area.

[0015] In some possible implementations, the target image area may correspond to a fixed imaging area of ​​the camera assembly.

[0016] In some possible implementations, the target image area may be a target imaging area determined according to a preset judgment condition.

[0017] In a possible implementation, the method further includes: updating the target image area in response to user input. In an embodiment of the present application, an interactive function of user input may be set. In response to user input, the target image area is re-determined according to the user input, and the image processing method is performed according to the determined target image area. In an embodiment of the present application, the electronic device may adjust the target image area for image processing according to the user input. Through this implementation, the user experience may be enhanced, the flexibility of image processing may be improved, etc.

[0018] In a second aspect, an embodiment of the present application further provides an electronic device, which includes a camera assembly and a processor. Wherein: the camera assembly is used to: shoot a first area based on a first focal length to obtain a first image. The processor is used to: input the first image, and perform a first operation according to a first mapping relationship, a first focus motor value and a first focal length; the first mapping relationship is a mapping relationship between the focus motor value of the camera assembly and the image distance; the first focus motor value is a theoretical focus motor value of the camera assembly relative to the target image area in the first image. The first operation includes at least one of the following: outputting a depth image of the target image area; outputting a neural network operation result of the target image area; outputting a control instruction, and the control instruction is used to instruct the electronic device to perform a certain device operation on the target image area.

[0019] In a possible implementation, the processor is specifically configured to: obtain a target image distance according to the first mapping relationship and the first focus motor value; the target image distance is the distance between the lens of the camera assembly and the image of the target image area. Obtain a target object distance according to the target image distance and the first focal length; the target object distance is the distance between the target image area of ​​the first image data and the camera assembly.

[0020] In a possible implementation, the processor is further configured to: obtain a first focus motor value according to a contrast ratio of the target image area in the first image data.

[0021] In a possible implementation, the camera assembly is integrated with a phase difference detection function. The processor is further configured to obtain a first focus motor value according to a phase difference of a target image area in the first image data.

[0022] In a possible implementation, the processor is specifically configured to: obtain a motor offset according to a phase difference of a target image area in the first image data; the motor offset is a difference between a current focus motor value of the camera assembly and a first focus motor value; and obtain the first focus motor value according to the current focus motor value and the motor offset.

[0023] For example, the motor offset may be calculated based on the phase difference and a defocus conversion coefficient (DCC) of the focus motor.

[0024] For example, the motor offset may be obtained based on the phase difference through artificial intelligence auto focus (AIAF) technology.

[0025] In a possible implementation, any of the above device operations includes: adjusting shooting parameters of a camera component.

[0026] In some possible implementations, the target image area may correspond to a fixed imaging area of ​​the camera assembly.

[0027] In some possible implementations, the target image area may be a target imaging area determined according to a preset judgment condition.

[0028] In a possible implementation, the processor is further configured to: update the target image area in response to user input.

[0029] In some possible implementations, the electronic device may be an image processing device, a smart home device, a video transmission interaction device, a detection device, etc. For example, it may be an image data acquisition processor, a projector, a sweeping robot, a smart phone, a desktop computer, a laptop computer, a tablet computer, a webcam, a detection camera, a face recognition door lock, etc.

[0030] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium comprising instructions. When the instructions are executed on a processor, the processor executes the image processing method as described in the first aspect above.

[0031] For the description of the technical principles and beneficial effects of the second and third aspects, reference can be made to the relevant content of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0033] Figure 2 A schematic diagram of the structure of a first electronic device provided in an embodiment of the present application;

[0034] Figure 3 A schematic diagram of the structure of a second electronic device provided in an embodiment of the present application;

[0035] Figure 4 A schematic diagram of the structure of a third electronic device provided in an embodiment of the present application;

[0036] Figure 5 A schematic diagram of the structure of a fourth electronic device provided in an embodiment of the present application;

[0037] Figure 6 A schematic structural diagram of a fourth camera assembly provided in an embodiment of the present application;

[0038] Figure 7 A schematic structural diagram of yet another fourth camera assembly provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of an image processing method provided in an embodiment of the present application Figure 1 ;

[0040] Fig. 9 Another image processing method provided in the present invention is shown in FIG. Figure 2 ;

[0041] Fig.10 A schematic diagram of another image processing method provided in the embodiment of the present application Figure 3 . DETAILED DESCRIPTION

[0042] It should be noted that the terms "first", "second", etc. involved in the embodiments of the present application are only used to distinguish features of the same type and cannot be understood as indicating relative importance, quantity, order, etc.

[0043] The terms "exemplary" or "for example" and the like in the embodiments of the present application are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the related concepts in a specific way.

[0044] The terms "coupling" and "connection" involved in the embodiments of the present application should be understood in a broad sense. For example, they may refer to a direct physical connection, or an indirect connection achieved through electronic devices, such as a connection achieved through resistors, inductors, capacitors or other electronic devices.

[0045] A depth image, also known as a range image, refers to an image that uses the distance (i.e., depth) from the image collector to each point in the scene as a pixel value. It directly reflects the geometric shape of the visible surface of the scene. Depth images can be used to easily solve problems in the description of three-dimensional targets. Depth images are widely used in image analysis, image processing, and various neural network algorithm processing based on image analysis and image processing. With the development of electronic information automation technology, more and more intelligent electronic devices are integrated with camera components. The target area is photographed by the camera component to obtain a photographed image of the target area. According to the algorithm processing of the photographed image, a depth image of the target area can be obtained. The depth image includes the object distance information of the target area. According to the object distance information contained in the depth image, the intelligent electronic device can perform different algorithm processing to obtain the corresponding algorithm processing results. Perform various intelligent operations according to the algorithm processing results. The application of depth images has greatly improved the operational intelligence of various intelligent electronic devices.

[0046] The present application embodiment provides an electronic device, such as Figure 1 As shown, the electronic device 1000 includes a processor 100 and a camera assembly 200. The processor 100 is coupled to the camera assembly 200. The camera assembly 200 is used to: photograph the first area Z1 to obtain a photographed image of the first area Z1. The processor 100 is used to: obtain a depth image of the first area Z1 based on the photographed image. In different application scenarios, the electronic device 1000 can perform different neural network operations based on the depth image to perform different intelligent operations.

[0047] In some possible implementations, such as Figure 1 The electronic device 1000 shown may be a first electronic device that obtains a depth image based on structured light. Figure 2As shown, the first electronic device 1000A includes a first processor 100A, a first camera assembly 200A and a structured light source 300A. The first camera assembly 200A is a binocular camera. Among them, the structured light source 300A is used to: emit a structured light signal to the first area Z1. The structured light signal is used to image feature points on the structural surface of the first area Z1. The first camera assembly 200A is used to: shoot the first area Z1 to obtain binocular image data about the first area Z1. The first processor 100A is used to: extract and calculate the feature point information in the binocular image data to obtain a depth image of the first area Z1. Perform neural network operations on the depth image, output the neural network operation results, or perform intelligent operations based on the neural network operation results. However, in this application, Figure 2 In the illustrated embodiment, it is necessary to set up a first camera assembly 200A as a binocular camera and a structured light source 300A, so that the device cost of the first electronic device 1000A is relatively high. In addition, the first processor 100A needs to process the binocular image data obtained by the binocular camera, extract and analyze the feature points in the binocular image data, etc., so as to generate a depth image. In this embodiment, the data processing amount of the first processor 100A is relatively large.

[0048] In some possible implementations, such as Figure 1 The electronic device 1000 shown may be a second electronic device that obtains a depth image based on a range finder. Figure 3 As shown, the second electronic device 1000B includes a second processor 100B, a second camera assembly 200B and a rangefinder 300B. The second camera assembly 200B is used to: photograph the first area Z1 to obtain image data about the first area Z1. The rangefinder 300B is used to: measure the distance of the first area Z1 to obtain distance information. The second processor 100B is used to: obtain a depth image based on the image data and the distance information. Perform a neural network operation on the depth image, output the neural network operation result, or perform intelligent operations based on the neural network operation result.

[0049] Exemplarily, the rangefinder 300B may be a rangefinder based on time of flight (ToF), etc.

[0050] However, in this application Figure 3In the illustrated embodiment, a rangefinder 300B is required to measure the distance of the first area Z1. The cost of the rangefinder 300B used in the field of image processing is usually very high, which greatly increases the cost of the second electronic device 1000B, which is not conducive to the promotion and application of the product. In addition, the second processor 100B needs to fuse and analyze the distance information obtained by the rangefinder 300B and the image data obtained by the second camera assembly 200B to obtain a depth image of the first area Z1. Under this processing method, the data processing volume of the second processor 100B is also relatively large.

[0051] In some possible implementations, such as Figure 1 The electronic device 1000 shown may be a third electronic device that can perform auto focus (PDAF) detection based on infrared phase detection. Figure 4 As shown, the third electronic device 1000C includes a third processor 100C and a third camera assembly 200C. The third camera assembly 200C is a binocular camera assembly with an integrated PDAF function. Among them, the third camera assembly 200C is used to: obtain multiple camera images with phase difference information. The third processor 100C is used to: perform neural network convolution processing on multiple camera images with phase difference information to obtain visual difference information of multiple camera images. A depth image is obtained based on the visual difference information. In an embodiment of the present application, the cost of devices such as rangefinders is reduced, but two or more camera sensing modules still need to be integrated in the third camera assembly 200C, and the device cost is relatively high. And using a convolution algorithm to calculate a depth image requires a lot of computing power overhead, which has a large computing power requirement for the third processor 100C.

[0052] In some possible implementations, in order to obtain a depth image while reducing the device cost and processing load of the electronic device 1000, as shown in FIG. Figure 1 The electronic device 1000 shown may be a fourth electronic device based on data relationship conversion. Figure 5 As shown, the fourth electronic device 1000D includes a fourth processor 100D and a fourth camera assembly 200D. Among them, the fourth camera assembly 200D is used to: shoot the first area Z1 based on the first focal length to obtain a first image. The fourth processor 100D is used to: input the first image, and perform a first operation according to the first mapping relationship, the first focus motor value and the first focal length; the first mapping relationship is the mapping relationship between the focus motor value of the fourth camera assembly 200D and the image distance; the first focus motor value is the theoretical focus motor value of the fourth camera assembly 200D relative to the target image area in the first image. According to the application scenario of the fourth electronic device, the fourth processor 100D can perform different first operations according to the calculation results:

[0053] Exemplarily, the first operation is: the fourth processor 100D outputs a depth image about the target image area. In the embodiment of the present application, the fourth electronic device 1000D is a device for calculating a depth image, and after calculating and obtaining the depth image about the target image area, it can directly output the depth image.

[0054] Exemplarily, the first operation is: the fourth processor 100D outputs the neural network operation result about the target image area. In the embodiment of the present application, the fourth electronic device 1000D is a device that needs to perform some neural network operations and judgments and identifications based on the depth image. At this time, the depth image is the intermediate processing data in the processing process of the fourth electronic device 1000D. The fourth processor 100D performs neural network operation processing based on the depth image, obtains the neural network operation result, and outputs the neural network operation result.

[0055] Exemplarily, the first operation is: the fourth processor 100D outputs a control instruction to the fourth camera assembly 200D, and the control instruction is used to control the fourth electronic device 1000D to perform a certain device operation on the target image area. In some examples, any device operation includes: adjusting the shooting parameters of the fourth camera assembly 200D. In an embodiment of the present application, the fourth electronic device 1000D is a device that needs to perform some intelligent operations based on the depth image. Some intelligent operations of the fourth electronic device 1000D need to be based on the depth image of the target image area. At this time, the depth image is the intermediate processing data in the processing process of the fourth electronic device 1000D. The fourth processor 100D needs to generate a control instruction for performing a certain device operation based on the depth image, or the fourth processor 100D needs to perform a neural network operation based on the depth image to obtain a neural network operation result, and obtain a control instruction based on the neural network operation result.

[0056] In this application as mentioned above Figure 5 In the illustrated embodiment, after the fourth processor 100D acquires the first image captured by the fourth camera assembly 200D, the object distances of each target in the first area Z1 can be obtained by performing data relationship conversion according to the first mapping relationship, the first focus motor value, and the first focal length, thereby generating a depth image of the target image area. Among them, the first mapping relationship is a mapping characteristic inherent to each camera assembly, and the first focal length is also a known parameter that can be acquired by the fourth processor 100D. The conversion relationship between the first mapping relationship, the first focus motor value, and the first focal length is also an inherent conversion relationship. The fourth processor 100D can quickly calculate the depth image based on multiple known parameters and mapping conversion relationships. In this way, without adding an additional structured light source 300A or a rangefinder 300B, there is no need to set up a binocular camera assembly to perform analysis and processing based on multiple images, and there is no need to obtain a depth image based on a complex neural network algorithm. Compared to the above, Figure 2 , Figure 3 and Figure 4 The embodiment shown, Figure 5 The embodiment shown can obtain a depth image on the basis of reducing device cost and data processing volume. At the same time, a neural network algorithm can be processed based on the depth image, or the fourth electronic device 1000D can be controlled to perform some intelligent operations based on the depth image.

[0057] In some possible implementations, such as Figure 6 As shown, the fourth camera assembly 200D includes a driving circuit 210D, a focus motor 220D and an imaging lens 230D. The driving circuit 210D outputs focus currents of different magnitudes to the focus motor 220D according to different focus motor values. The focus motor 220D is driven according to the focus currents of different magnitudes, so that the focus motor 220D adjusts the position of the imaging lens 230D to adjust the image distance of the imaging lens 230D with respect to the image. Therefore, according to the structural design, parameter design, etc. of different camera assemblies, there are corresponding first mapping relationships, and the first mapping relationship corresponds to the mapping relationship between different focus motor values ​​and image distances of the camera assembly.

[0058] For example, Figure 6 The driving circuit 210D shown is a digital-to-analog converter. The digital-to-analog converter performs digital-to-analog conversion on the focus motor value to obtain the focus current. At this time, the first mapping relationship corresponding to the camera component can be reflected according to the conversion accuracy, conversion bit number, and design of the reference analog signal source of the digital-to-analog converter.

[0059] In some possible implementations, such as Figure 7 As shown, the fourth camera assembly 200D may further include a zoom motor 240D. The zoom motor 240D is used to adjust the imaging lens 230D to adjust the focal length of the fourth camera assembly 200D. In some scenarios, the fourth camera assembly 200D is a camera assembly without a zoom function, in which case the first focal length is a fixed value. Figure 7 In the illustrated scene, the fourth camera assembly 200D is a camera assembly with a zoom function. The fourth processor 100D outputs a focus adjustment instruction to the fourth camera assembly 200D, and adjusts the focus of the fourth camera assembly 200D according to the focus adjustment instruction. The adjusted first focus is also a known parameter of the fourth processor 100D.

[0060] In some possible implementations, the fourth processor 100D is specifically configured to: obtain a target object distance according to the target image distance and the first focal length; the target object distance is the distance between the target image area of ​​the first image and the fourth camera assembly 200D. Obtain the target object distance of the first image with respect to the target image area according to the target image distance and the first focal length. Perform the first operation according to the target object distance.

[0061] In some possible implementations, the first focus motor value can be calculated based on contrast detection autofocus (CDAF). At this time, the fourth processor 100D is also used to obtain the first focus motor value according to the contrast of the target image area in the first image. In the embodiment of the present application, there is no need to set or modify additional devices on the fourth electronic device 1000D, and the first focus motor value can be directly obtained through CDAF technology.

[0062] In some possible implementations, the first focus motor value can be calculated based on phase detection auto focus technology (PDAF). At this time, the fourth camera assembly 200D is integrated with a phase difference detection function; the fourth processor 100D is also used to: obtain the first focus motor value according to the phase difference of the target image area in the first image. In the embodiment of the present application, the fourth camera assembly 200D needs to be integrated with a phase difference detection function, so that PDAF processing can be implemented based on the first image taken by the fourth camera assembly 200D to obtain the first focus motor value. But compared to Figure 4 In the embodiment shown, the embodiment of the present application only requires a single camera sensing device.

[0063] Exemplarily, the fourth processor 100D is specifically used to: obtain a motor offset according to a phase difference of the target image area in the first image; the motor offset is the difference between the current focus motor value of the fourth camera assembly 200D and the first focus motor value. The first focus motor value is obtained according to the current focus motor value and the motor offset. In the embodiment of the present application, the difference between the current focus motor value and the first focus motor value can be obtained based on the PDAF technology. The first focus motor value can be obtained according to the difference and the current focus motor value.

[0064] Exemplarily, the phase difference of the target image area of ​​the first image may be calculated, and the motor offset may be obtained based on the phase difference and the defocus conversion rate of the fourth camera assembly 200D.

[0065] Exemplarily, the phase difference of the target image area of ​​the first image may be calculated, and the motor offset may be obtained based on an artificial intelligence auto focus (AIAF) technology.

[0066] In some possible implementations, the fourth processor 100D is further configured to: update the target image area in response to user input. In the embodiment of the present application, the fourth electronic device 1000D may also update the target image area according to user input, and perform the first operation process based on the target image area.

[0067] In some possible implementations, the fourth electronic device 1000D may be an image processing device, a smart home device, a video transmission interaction device, a detection device, etc. For example, it may be an image data acquisition processor, a projector, a sweeping robot, a smart phone, a desktop computer, a laptop computer, a tablet computer, a webcam, a detection camera, a face recognition door lock, etc.

[0068] Based on the above Figure 5 , Figure 6 and Figure 7 The fourth electronic device 1000D of the structure shown in the figure can perform the following operations: Figure 8 The image processing method of step S100-step S200 shown in FIG.

[0069] S100 , the fourth camera assembly 200D photographs the first area Z1 based on the first focal length to obtain a first image.

[0070] In the embodiments of the present application, Figure 5 As shown, the fourth camera assembly 200D photographs the first area Z1 to obtain a first image of the first area Z1. Then the fourth camera assembly 200D outputs the first image to the fourth processor 100D.

[0071] In some possible implementations, such as Figure 6 As shown, the fourth camera assembly 200D is a camera assembly without a zoom function. At this time, the first focal length is a fixed parameter value.

[0072] In some possible implementations, such as Figure 7 As shown, the fourth camera assembly 200D is a camera assembly with a zoom function. At this time, the first interaction is an adjustable parameter value. However, in each process of adjusting the focal length of the fourth camera assembly 200D, the fourth processor 100D outputs a focal length adjustment instruction to the fourth camera assembly 200D. The focal length of the fourth camera assembly 200D is adjusted by the focal length adjustment instruction. Then the adjusted first focal length is still a known parameter for the fourth processor 100D.

[0073] S200: Input a first image, and perform a first operation according to a first mapping relationship, a first focus motor value, and a first focal length.

[0074] In some possible implementations, the target object distance of the fourth camera assembly 200D can be obtained according to the first mapping relationship and the first focus motor value. The first focus motor value is a theoretical focus motor value of the fourth camera assembly 200D relative to the target image area in the first image. The target image distance is the distance between the lens of the fourth camera assembly 200D and the image of the target image area. Then, according to the target image distance of the fourth camera assembly 200D and the first focal length, the target object distance about the target image area is obtained. According to the target object distance, a depth image can be obtained.

[0075] In some possible implementations, step S200 may include the following: Fig. 9 The sub-operations shown include step S210-step S230:

[0076] S210: Obtain a target image distance according to a first mapping relationship and a first focus motor value.

[0077] For example, the first mapping relationship is an inherent mapping characteristic of each camera component. When the camera component is initially designed, the corresponding focus motor value is designed to correspond to different imaging distances. Figure 6 As shown, different focus motor values ​​are represented in the form of digital signals. By performing digital-to-analog conversion on the focus motor value of the digital signal through the driving circuit 210D, a focus current in the form of an analog signal can be obtained. Focus motor values ​​of different values ​​can obtain corresponding different focus currents. By driving the imaging lens 230D with focus currents of different sizes, different image distances can be obtained. Therefore, for each camera component, we can obtain a mapping relationship between the target image distance and the first focus motor value. This mapping relationship may be a linear mapping relationship or a nonlinear mapping relationship.

[0078] In some possible implementations, step S210 may include: Fig.10 The sub-operations shown include step S211-step S212:

[0079] S211 , determining a first focus motor value.

[0080] In some possible implementations, the first focus motor value can be obtained based on the contrast of the target image area in the first image. Exemplarily, the fourth processor 100D can calculate the first focus motor value based on contrast detection auto focus (CDAF). In an embodiment of the present application, CDAF technology is a camera auto focus technology that determines the position of the focus by analyzing the contrast of the image. Compared with traditional phase focusing technology, CDAF technology has higher accuracy and a wider focus range. The advantage of CDAF technology is that it can achieve higher focus accuracy and a wider focus range, while also achieving smooth focus transition in video shooting. After the target image area is selected in the first image, the target image area is used as the target focus area, and the first focus motor value of the quasi-focus can be determined by CDAF technology. The calculation formula for the first focus motor value can be expressed in the form of the following formula (1):

[0081] code = func0(CDAF) (1)

[0082] Wherein, code is the first focus motor value, and the func0(CDAF) function is used to represent the mapping calculation of the focus motor value obtained based on the CDAF technology.

[0083] In the embodiment of the present application, the first focus motor value is calculated based on the CDAF technology, which expands the application of the CDAF technology. At the same time, the CADF technology can be implemented based on ordinary camera sensing components without adding additional hardware. Figure 2 , Figure 3 and Figure 4 The embodiment shown greatly reduces the hardware cost.

[0084] In some possible implementations, the first focus motor value may be obtained according to the phase difference of the target image area in the first image. For example, the fourth processor 100D may calculate the first focus motor value based on phase detection auto focus (PDAF) technology:

[0085] First, the fourth processor 100D divides the first image into a plurality of sub-image regions and calculates phase differences (PD) of the plurality of sub-image regions based on the PDAF technology.

[0086] Then, the motor offset is obtained according to the phase difference of the target image area in the first image; the motor offset is the difference between the current focus motor value of the fourth camera assembly 200D and the first focus motor value. The phase difference of the sub-image area corresponding to the target image area is adjusted to be close to 0, so that the target image area becomes clearer. The motor offset can be calculated in different ways:

[0087] In some examples, the motor offset may be calculated based on the phase difference and the defocus conversion coefficient (DCC) of the focus motor. The specific calculation may be expressed in the form of the following formula (2):

[0088] △code = DCC * PD + offset (2)

[0089] In the formula, △code is the motor offset, DCC is the defocus conversion rate of the fourth camera assembly 200D, PD is the phase difference corresponding to the target image area, and offset is the deviation in actual use. Depending on the usage scenario, offset can be positive, negative or zero.

[0090] In some examples, after the phase difference of the target image area is calculated, the motor offset can be obtained by artificial intelligence auto focus (AIAF) technology. The specific calculation can be expressed in the form of the following formula (3):

[0091] △code = func1(PD) (3)

[0092] Where △code is the motor offset, func1() is used to represent the AIAF calculation, and PD is the phase difference of the target image area.

[0093] Finally, the first focus motor value is obtained according to the motor offset and the current focus motor value. The specific calculation can be expressed in the form of the following formula (4):

[0094] code = △code + code_current (4)

[0095] Where △code is the motor offset, code_curren is the current focus motor value, and code is the first focus motor value.

[0096] In the embodiment of the present application, the phase difference is calculated based on the PDAF technology, and the first focus motor value is obtained based on the phase difference. Based on this implementation, the application of the PDAF technology is expanded. At the same time, this implementation only requires a single camera sensing module with a phase detection function to be set in the fourth camera assembly 200D. Figure 2 , Figure 3 and Figure 4 The embodiment shown reduces the device cost.

[0097] S212: Obtain a target image distance according to the first mapping relationship and the first focus motor value.

[0098] In the embodiment of the present application, according to the first mapping relationship designed by the fourth camera assembly 200D at the factory, after obtaining the first focus motor value, a conversion process is performed based on the first mapping relationship to obtain the target image distance. The conversion formula can be expressed in the form of the following formula (5):

[0099] v = func2(code) (5)

[0100] Wherein, v is the target image distance between the lens of the fourth camera assembly 200D and the image of the target imaging area, the func2() function is used to represent the calculation of the first mapping relationship between different focus motor values ​​and image distances, and code is the first focus motor value.

[0101] S220: Obtain the target object distance according to the first focal length and the target image distance.

[0102] For example, after obtaining the target image distance according to the first focus motor value, since the first focal length is a known parameter of the fourth processor 100D, the fourth processor 100D can obtain the target object distance based on the first focal length and the target image distance. Regarding camera imaging technology, there is a lens Gaussian formula, which is specifically shown in the following formula (6):

[0103] 1 / f = 1 / u -1 / v (6)

[0104] Wherein, f is the first focal length of the fourth camera assembly 200D, v is the target image distance between the lens of the fourth camera assembly 200D and the image of the target imaging area, and u is the target object distance between the fourth camera assembly 200D and the target imaging area.

[0105] After the target object distance is calculated, the target object distance is the component information of the depth image. According to the target object distance, the depth image of the target imaging area can be obtained.

[0106] S230: Execute a first operation.

[0107] In some possible implementations, different first operations may be performed according to different application scenarios of the fourth electronic device 1000D:

[0108] In some examples, the fourth electronic device 1000D is a dedicated image processing device for generating a depth image. At this time, after calculating the depth image of the target image area according to the first mapping relationship, the first focus motor value and the first focal length, the fourth electronic device 1000D outputs the depth image.

[0109] In some examples, the fourth electronic device 1000D is an electronic device for intelligent image recognition. At this time, after calculating the depth image of the target image area according to the first mapping relationship, the first focus motor value and the first focal length, the fourth electronic device 1000D outputs the neural network algorithm result processed by the neural network algorithm according to the depth image.

[0110] Taking the fourth electronic device 1000D as a camera as an example, the camera can detect the scene in the first area Z1. When an abnormal event (such as fire, smoke, abnormal behavior of personnel, etc.) occurs in the first area Z1, the area where the abnormal event occurs can be used as the target image area, and the depth image of the target image area is calculated based on the method recorded in the above step embodiment, and the neural network algorithm is processed based on the depth image to obtain the neural network operation result. The neural network operation result can be: output alarm, output abnormality recognition, etc.

[0111] Taking the fourth electronic device 1000D as a mobile phone as an example, the target image area in the first area Z1 can be detected to obtain a depth image, and a neural network algorithm is processed according to the depth image to obtain a neural network operation result of the target image area. The neural network operation result can be a category analysis of the photographed object, etc.

[0112] The embodiments of the present application do not limit the application scenarios of neural network operations. All devices that use camera components to generate depth images and perform neural network operations based on the depth images can use the methods described in the embodiments of the present application to output the operation results after the neural network operations.

[0113] In some examples, the fourth electronic device 1000D is an intelligent electronic device based on the depth image. At this time, the fourth electronic device 1000D needs to perform some intelligent operations based on the processing and analysis of the depth image.

[0114] Exemplarily, taking the fourth electronic device 1000D as a mobile phone, when a mobile phone is used for online live broadcast, there may be human activities in the first area Z1 where the live broadcast scene is located, or there may be multiple people at different distances. The target image area can be determined according to the preset judgment conditions. For example, the area where the person closest to the lens is located is used as the target image area, or the area where the person whose movement amplitude is within a certain range is located is used as the target image area. After the target image area is set, the shooting parameters such as the matching focal length and matching image distance of the target image area can be determined according to the depth image of the target image area. The fourth processor 100D can output a control instruction to the fourth camera assembly 200D according to the depth image, and adjust the shooting parameters of the fourth camera assembly 200D according to the control instruction to improve the imaging quality of the target image area.

[0115] Exemplarily, taking the fourth electronic device 1000D as a sweeping robot, the sweeping robot scans the first area Z1 and identifies an attention event in the first area Z1. The location of the attention event is taken as the target image area, and a depth image of the target image area is obtained according to the scheme described in the above embodiment. The neural network algorithm is used to identify the depth image to obtain the neural network operation result. The fourth processor 100D outputs a control instruction according to the neural network operation result. The control instruction is used to instruct the fourth electronic device 1000D to perform corresponding device operations, such as cleaning the target image area, or adjusting the cleaning mode for cleaning the target image area, or changing the new route to bypass the target image area.

[0116] Exemplarily, taking the fourth electronic device 1000D as a projector, the projector shoots the projection surface, obtains a depth image according to the object distance of the projection surface, and determines the projection focal length of the projection according to the depth image. The fourth processor 100D can output a control instruction according to the determined projection focal length to control the projection light component of the projector to project the picture based on the determined projection focal length to improve the projection quality. Alternatively, the fourth processor 100D can perform neural network algorithm processing according to the depth image, calculate whether the projection surface is deformed, etc., and adjust the projection parameters of the projection image data according to the deformation of the projection surface to eliminate the imaging distortion of the emitted projection light on the projection surface.

[0117] Exemplarily, the fourth electronic device 1000D may also perform living body (or human body, human face) recognition on the target image area within the first area Z1, and perform human-computer interaction or provide personalized services according to the recognition result.

[0118] In actual applications, the first operation may be one or more of the above processing types. For example, taking the fourth electronic device 1000D as a face recognition door lock, the face recognition door lock can output the depth image of the identified person to the memory for label classification storage based on the generated depth image to assist in subsequent recognition algorithm processing, and can display the face recognition result obtained based on the neural network operation processing of the depth image on the display screen, and can also control the door lock to open or close according to the face recognition result.

[0119] In some possible implementations, the target image area may correspond to a fixed imaging area of ​​the fourth camera assembly 200D.

[0120] In some possible implementations, the target image area may be a target imaging area determined by the fourth processor 100D according to a preset judgment condition.

[0121] In some possible implementations, the target image region may be a region adjusted and updated by a user. In this case, the image processing method further includes: updating the target image region in response to user input. For example, Figure 5 As shown, the fourth processor 100D responds to the user input, re-determines the target image area according to the user input, and performs the above-mentioned image processing method according to the determined target image area. In the embodiment of the present application, the fourth electronic device 1000D can adjust the target image area for image processing according to the user input. Through this implementation, the user experience can be enhanced, the flexibility of image processing can be improved, etc.

[0122] The embodiment of the present application provides an image processing method and an electronic device, which includes a camera component. The image processing method is implemented based on the electronic device, and the specific method is: the first area Z1 is photographed by the camera component based on the first focal length to obtain a first image. According to the first focus motor value of the target image area and the image distance-focus motor value mapping relationship of the camera component, the target image distance of the target image area is obtained. The target object distance is calculated according to the target image distance and the first focal length. The depth image of the target image area can be obtained by the target object distance. Alternatively, a neural network operation is performed based on the depth image to obtain a neural network operation result. Alternatively, a control instruction is output based on the depth image to control the electronic device to perform related intelligent operations. The embodiment of the present application does not need to set up additional measuring instruments or more expensive camera components, and the depth image can be obtained with a universal device. At the same time, in the process of generating the depth image, a large amount of data calculation is not required, and the depth image can be generated based on known parameters and / or easily obtained parameters. Compared with the traditional solution that requires a large amount of calculation to generate the depth image, the present solution can realize the calculation of the depth image in a cheaper and lighter electronic device, and various artificial intelligence solutions are applied based on the depth image. This solution greatly reduces the cost and data processing volume of various electronic devices based on depth images, allowing such electronic equipment products to be generated, applied and popularized more quickly.

[0123] The present application also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a processor, the processor executes the image processing method described in the above embodiment (for example, Figure 8 , Fig. 9 and Fig.10 The image processing method described herein).

[0124] The processor involved in the embodiments of the present application may be a chip. For example, it may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0125] The memory involved in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0126] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0127] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0130] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one device or distributed on multiple devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0131] In addition, each functional module in each embodiment of the present application may be integrated into one device, or each module may exist physically separately, or two or more modules may be integrated into one device.

[0132] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing a computer program instruction on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may contain one or more servers, data centers and other data storage devices that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0133] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Based on an electronic device; the electronic device includes a camera assembly; the method includes: Controlling the camera assembly to photograph a first area based on a first focal length to obtain a first image; The first image is input, and a first operation is performed according to a first mapping relationship, a first focus motor value, and the first focal length; the first mapping relationship is a mapping relationship between the focus motor value of the camera component and the image distance; the first focus motor value is a theoretical focus motor value of the camera component relative to a target image area in the first image; and the first operation includes at least one of the following: Outputting a depth image of the target image area; Outputting a neural network operation result on the target image area; Output a control instruction, where the control instruction is used to instruct the electronic device to perform a certain device operation on the target image area.

2. The image processing method according to claim 1, characterized in that: The performing a first operation according to the first mapping relationship, the first focus motor value and the first focal length includes: According to the first mapping relationship and the first focus motor value, a target image distance is obtained; the target image distance is the distance between the lens of the camera assembly and the image of the target image area; Obtaining a target object distance according to the target image distance and the first focal length; the target object distance is the distance between the target image area of ​​the first image and the camera assembly; The first operation is performed according to the target object distance.

3. The image processing method according to claim 2, characterized in that: The method further comprises: The first focus motor value is obtained according to the contrast of the target image area in the first image.

4. The image processing method according to claim 2, characterized in that: The method further comprises: The first focus motor value is obtained according to the phase difference of the target image area in the first image.

5. The image processing method according to claim 4, characterized in that: The obtaining the first focus motor value according to the phase difference of the target image area in the first image includes: Obtaining a motor offset according to a phase difference of the target image area in the first image; the motor offset being a difference between a current focus motor value of the camera assembly and the first focus motor value; The first focus motor value is obtained according to the current focus motor value and the motor offset.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The certain device operation includes: adjusting the shooting parameters of the camera component.

7. The image processing method according to any one of claims 1 to 6, characterized in that: The method further comprises: In response to user input, the target image area is updated.

8. An electronic device, characterized in that: Comprising a camera assembly and a processor; wherein: The camera assembly is used to: photograph a first area based on a first focal length to obtain a first image; The processor is used to: The first image is input, and a first operation is performed according to a first mapping relationship, a first focus motor value, and the first focal length; the first mapping relationship is a mapping relationship between the focus motor value of the camera component and the image distance; the first focus motor value is a theoretical focus motor value of the camera component relative to a target image area in the first image; and the first operation includes at least one of the following: Outputting a depth image of the target image area; Outputting a neural network operation result on the target image area; Output a control instruction, where the control instruction is used to instruct the electronic device to perform a certain device operation on the target image area.

9. The electronic device according to claim 8, characterized in that: The processor is specifically used for: According to the first mapping relationship and the first focus motor value, a target image distance is obtained; the target image distance is the distance between the lens of the camera assembly and the image of the target image area; The target object distance is obtained according to the target image distance and the first focal length; the target object distance is the distance between the target image area of ​​the first image and the camera component.

10. The electronic device according to claim 9, characterized in that: The processor is further configured to: The first focus motor value is obtained according to the contrast of the target image area in the first image.

11. The electronic device according to claim 9, characterized in that: The camera assembly is integrated with a phase difference detection function; and the processor is further used for: The first focus motor value is obtained according to the phase difference of the target image area in the first image.

12. The electronic device according to claim 11, characterized in that: The processor is specifically used for: Obtaining a motor offset according to a phase difference of the target image area in the first image; the motor offset being a difference between a current focus motor value of the camera assembly and the first focus motor value; The first focus motor value is obtained according to the current focus motor value and the motor offset.

13. The electronic device according to any one of claims 8 to 12, characterized in that: Any of the device operations includes: adjusting the shooting parameters of the camera component.

14. The electronic device according to any one of claims 8 to 13, characterized in that: The processor is further configured to: In response to user input, the target image area is updated.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions; when the instructions are executed on a processor, the processor is enabled to execute the image processing method as described in any one of claims 1 to 7.

Images