Shooting method, device, electronic device and storage medium

Abstract

The present application discloses a shooting method, device, electronic device, and storage medium, belonging to the field of electronic technology. The method comprises: during the focusing process of a camera module, obtaining dynamic visual events of the image sensor of the camera module when capturing each image frame, the dynamic visual events including brightness change information of pixels; based on the brightness change information of the pixels in each dynamic visual event, determining the brightness change area of the image frame corresponding to the dynamic visual event; determining the camera module position corresponding to the target focus based on the brightness change areas of multiple image frames; and controlling the camera module to focus on the target focus based on the camera module position corresponding to the target focus.

Description

Technical Field

[0001] The present application belongs to the field of electronic technology, and specifically relates to a shooting method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of electronic technology, electronic devices such as smartphones and tablets are becoming increasingly popular and gradually becoming an indispensable part of people's daily lives. Among them, the shooting function is one of the important functions of electronic devices. Users can capture images through the camera module of electronic devices, so that they can capture every detail of their lives in real time.

[0003] During the capture process, the camera module of an electronic device typically focuses based on the position of the subject, thereby improving the clarity of the subject's image area in the captured image. However, currently, the subject's position may not be determined during focus, resulting in reduced focus accuracy and, in turn, lowered image quality. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a shooting method, device, electronic device and storage medium, which can solve the problem of reduced focus accuracy due to inaccurate focus position determination, thereby reducing the quality of the captured image.

[0005] In a first aspect, an embodiment of the present application provides a shooting method, which is applied to an electronic device including a camera module, and the method includes:

[0006] During the focusing process of the camera module, dynamic visual events of the image sensor of the camera module when capturing each image frame are obtained, and the dynamic visual events include brightness change information of the pixel points;

[0007] Determining the brightness change area of the image frame corresponding to the dynamic visual event based on the brightness change information of the pixel points in each dynamic visual event;

[0008] Determine the camera module position corresponding to the target focus based on the brightness change area of multiple image frames;

[0009] Based on the position of the camera module corresponding to the target focus, the camera module is controlled to focus on the target focus.

[0010] In a second aspect, an embodiment of the present application provides a photographing device, which is applied to an electronic device including a camera module, and the device includes:

[0011] A dynamic visual event acquisition module is used to acquire dynamic visual events generated by the image sensor of the camera module when capturing each image frame during the focusing process of the camera module. The dynamic visual events include brightness change information of at least some pixels;

[0012] a brightness change information determination module, configured to determine the brightness change area of the image frame corresponding to the dynamic visual event based on the brightness change information of the pixels in each dynamic visual event;

[0013] A position determination module is used to determine the camera module position corresponding to the target focus based on the brightness change area of multiple image frames;

[0014] The focusing module is used to control the camera module to focus on the target focus based on the camera module position corresponding to the target focus.

[0015] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method of the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method of the first aspect.

[0019] In an embodiment of the present application, during the focusing process of the camera module, the dynamic visual events of the image sensor of the camera module when capturing each image frame are first obtained; then, based on the brightness change information of the pixel points in each dynamic visual event, the brightness change area of the image frame corresponding to the dynamic visual event is determined; then, based on the brightness change area of multiple image frames, the camera module position corresponding to the target focus is determined; finally, based on the camera module position corresponding to the target focus, the camera module is controlled to focus on the target focus. In this way, the dynamic visual events obtained by the image sensor can be used to accurately determine the camera module position corresponding to the target focus, thereby improving the accuracy of focusing and further improving the quality of the captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flowchart of an embodiment of the shooting method provided by this application;

[0021] Figure 2 is a schematic diagram of the focusing process in an embodiment of the shooting method provided by the present application;

[0022] Figure 3is a schematic diagram of a target quadratic curve in an embodiment of the shooting method provided by this application;

[0023] Figure 4 Schematic diagram of foreground and background light spots under different spherical aberrations in an embodiment of the shooting method provided by the present application;

[0024] Figure 5 is a structural diagram of an embodiment of a photographing device provided by the present application;

[0025] Figure 6 is a schematic structural diagram of an embodiment of an electronic device provided by this application;

[0026] Figure 7 It is a structural diagram of another embodiment of an electronic device provided by the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0028] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] The following describes the shooting method provided in the embodiment of the present application in detail through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0030] In related technologies, the camera module of an electronic device generally implements autofocus using phase detection autofocus and contrast detection autofocus. The principles of these two methods both use the red, green, and blue (RGB) information in the image frame captured by the image sensor to calculate the focus position (also known as the "camera module position corresponding to the focus").

[0031] The method of calculating the focus position based on the RGB information in the image frame will have problems with focusing on point light source scenes. This is because the point light source (i.e., the photographed object) saturates the pixels, resulting in overexposed information with no or little detail. As a result, the focus position of the point light source cannot be calculated, resulting in reduced focus accuracy and, in turn, reduced image quality.

[0032] See Figure 1 , a flow chart of a shooting method provided in an embodiment of the present application, which is applied to an electronic device including a camera module. Figure 1 As shown, the above method includes the following steps 101 to 104.

[0033] Step 101: During the focusing process of the camera module, dynamic visual events of the image sensor of the camera module when capturing each image frame are obtained. The dynamic visual events include brightness change information of at least part of the pixels.

[0034] In the embodiment of the present application, the camera module focusing process may include at least one of a focusing process and a defocusing process. Specifically, the focusing process may include: the camera module moving from defocus to focus, and then moving from focus to defocus, that is, the camera module focusing process includes the focusing process and the defocusing process, so that the camera module position corresponding to the determined focus is more accurate, thereby improving the shooting quality.

[0035] The image sensor of the above-mentioned camera module can be a two-in-one sensor that can serve as a dynamic vision sensor (DVS) and a red, green, and blue (RGB) sensor at the same time. That is, the two-in-one sensor can not only sense RGB information to acquire image frames, but also realize the function of a dynamic vision sensor in the process of acquiring image frames.

[0036] The DVS is a bio-inspired sensor that captures dynamic scene information, mitigating data redundancy and latency. In DVS, all pixels are individually detected, and after each exposure for a short, fixed time, the sensor determines whether the pixel has a sufficiently large brightness change, that is, whether the brightness change exceeds the DVS threshold. If the brightness change of a pixel exceeds the DVS threshold, the pixel actively requests to be read (i.e., the pixel is read once if the brightness change does not exceed the DVS threshold once). If the brightness change of a pixel does not exceed the DVS threshold, the pixel is skipped and is not read.

[0037] In this way, all pixels operate independently, and only pixels with varying brightness are output. During the camera module's focusing process, the two-in-one sensor can capture the brightness variation of each pixel at any given moment, thereby capturing the dynamic visual events of each image frame captured by the sensor.

[0038] For example, Figure 2 As shown, when the electronic device shoots light, the motor of the camera module can push the lens module to move so that the camera module is in the focusing process or the defocusing process. For example, the motor pushes the lens module from near to far, so that the focus position moves from the focus to the sensor surface, and the spot area changes from large to small (such as Figure 2 As the motor drives the lens, the RGB Bayer array in the two-in-one sensor captures RGB information to form image frames, while the DVS real-sense pixels (i.e., pixels) capture brightness changes to form dynamic visual events.

[0039] The dynamic visual event includes brightness change information for each pixel at the time point of the corresponding image frame acquisition. The brightness change information indicates whether the brightness of the pixel increased, decreased, or remained unchanged. Specifically, the dynamic visual event may also include the coordinates of each pixel and the time the dynamic visual event was acquired, etc.

[0040] For example, when the motor starts to push the lens module to move, the current frame is acquired and the output event E(x, y, P, t) is output, where (x, y) represents the coordinates on the sensor, P is the polarity, and t is the output time. The subsequent movement times are recorded as: t1, t2, t3, .... Specifically, when the above-mentioned dynamic visual event corresponds to an image frame, t can be the time when the image frame is acquired;

[0041] Define the brightness change at position (x, y) as A, where an increase in brightness is represented by A>0, a decrease in brightness is represented by A<0, and no change in brightness is represented by A=0. Then, P can be expressed by the following formula (1).

[0042]

[0043] Step 102: Based on the brightness change information of the pixels in each dynamic visual event, determine the brightness change area of the image frame corresponding to the dynamic visual event.

[0044] In an embodiment of the present application, the brightness change area of the image frame corresponding to the dynamic visual event is determined based on the brightness change information of the pixel points in each dynamic visual event. It can be determined whether the brightness of each pixel point changes based on the brightness change information of each pixel point in the above dynamic visual event, and the number of pixels whose brightness changes is counted as the brightness change area of the image frame corresponding to the above dynamic visual event.

[0045] Specifically, the brightness change area S can be calculated by assuming that the resolution of the two-in-one sensor is m×n. Then, the brightness change area S can be calculated using the following formula (2).

[0046]

[0047] Step 103: Determine the camera module position corresponding to the target focus according to the brightness change area of the multiple image frames.

[0048] In the embodiment of the present application, the determination of the camera module position corresponding to the target focus based on the brightness variation area of the multiple image frames can be performed by fitting the brightness variation area of the multiple image frames to calculate the camera module position corresponding to the target focus. Specifically, when the electronic device determines the brightness variation area of the multiple image frames, the electronic device can fit the brightness variation area of the multiple image frames using a preset fitting function to calculate the camera module position corresponding to the target focus.

[0049] The above-mentioned preset fitting function can be any function that can fit the brightness change area of multiple image frames to calculate the camera module position corresponding to the above-mentioned target focus.

[0050] For example, the electronic device can fit the above-mentioned multiple image frames through a preset third-order fitting function to obtain a cubic function curve. The cubic function curve can represent the relationship between the brightness change area and the position of the lens of the camera module, and the electronic device can determine the lens position corresponding to the minimum value of the cubic function curve as the camera module position corresponding to the above-mentioned target focus.

[0051] The target focus may be any subject in the camera module's shooting preview interface, which may be automatically determined by the electronic device or selected based on user input. For example, the target focus may be a face determined by the camera module.

[0052] Step 104: Based on the position of the camera module corresponding to the target focus, control the camera module to focus on the target focus.

[0053] The above-mentioned control of the camera module to focus on the target focus based on the camera module position corresponding to the target focus can be understood as controlling the lens of the camera module to move to the camera module position corresponding to the above-mentioned target focus to achieve the focus of the camera module.

[0054] In an embodiment of the present application, during the focusing process of the camera module, the dynamic visual events of the image sensor of the camera module when capturing each image frame are first obtained; then, based on the brightness change information of the pixel points in each dynamic visual event, the brightness change area of the image frame corresponding to the dynamic visual event is determined; then, based on the brightness change area of multiple image frames, the camera module position corresponding to the target focus is determined; finally, based on the camera module position corresponding to the target focus, the camera module is controlled to focus on the target focus. In this way, the dynamic visual events obtained by the image sensor can be used to accurately determine the camera module position corresponding to the target focus, thereby improving the accuracy of focusing and further improving the quality of the captured image.

[0055] In some embodiments, determining the camera module position corresponding to the target focus based on the brightness change area of the multiple image frames includes:

[0056] Fit the camera module position and the brightness change area of the image frame to obtain the target quadratic curve;

[0057] The camera module position corresponding to the maximum value of the target quadratic curve is determined as the camera module position corresponding to the target focus.

[0058] In this embodiment, the target quadratic curve is fitted based on the brightness change area of multiple image frames, and the camera module position corresponding to the maximum value of the curve is determined as the camera module position corresponding to the target focus, so that the determined camera module position corresponding to the target focus is more accurate, further improving the focusing accuracy, and thus improving the shooting quality.

[0059] The above-mentioned fitting of the camera module position and the brightness change area of the image frame to obtain the target quadratic curve can be performed by fitting the camera module position and the brightness change area of the image frame through a preset quadratic function to obtain the target quadratic curve.

[0060] For example, during the focusing process, it is assumed that the electronic device obtains the brightness change areas S1, S2 and S3 of three image frames, and uses these three values to fit a quadratic function y=ax 2 +bx+c, the curve of the quadratic function (i.e. the target quadratic curve) is as follows Figure 2 As shown, the maximum value of the curve (the maximum value of the y-axis) is calculated, and the camera module position in the x-axis corresponding to the maximum value is determined as the camera module position corresponding to the above-mentioned target focus. If the curve of the quadratic function is the curve of the xy coordinate axis, the above-mentioned symmetry axis -b / 2a is the x-axis coordinate corresponding to the maximum value on the y-axis, and -b / 2a on the x-axis is determined as the camera module position corresponding to the above-mentioned target focus.

[0061] The above-mentioned control of the camera module to focus on the target focus based on the camera module position corresponding to the target focus can be that the electronic device obtains the current position of the camera module lens, compares the current position with the camera module position corresponding to the target focus, determines the movement direction of the lens, and moves the lens to the camera module position corresponding to the target focus according to the movement direction of the lens.

[0062] In some embodiments, after determining the brightness change area of the image frame corresponding to the dynamic visual event based on the brightness change information of the pixels in each dynamic visual event, the method further includes:

[0063] The focus state of the camera module is determined based on the change trend of the brightness change area of multiple image frames. The change trend is used for the brightness change area to increase or decrease. The focus state includes the focused state or the defocused state.

[0064] The above-mentioned controlling the camera module to focus on the target focus based on the camera module position corresponding to the target focus may include:

[0065] Based on the focusing state and the camera module position corresponding to the target focus, the camera module is controlled to focus on the target focus.

[0066] In this embodiment, the focusing state of the camera module is determined by the changing trend of the brightness change area of multiple image frames, and the focus of the camera module is controlled according to the focusing state and the camera module position corresponding to the target focus, so that the camera module can be moved to the camera module position corresponding to the target focus more quickly, thereby improving the focusing efficiency.

[0067] For example, the brightness change area of the above-mentioned multi-frame image frame includes the above-mentioned S1, S2 and S3, and as Figure 3 As shown, if S1, S2 and S3 decrease in sequence, the electronic device determines that the camera module is in a focused state, and the electronic device controls the camera module to continue moving in the focusing direction until it reaches the camera module position corresponding to the target focus; conversely, if S1, S2 and S3 increase in sequence, the electronic device determines that the camera module is in a defocused state, and the electronic device controls the camera module to move in the opposite direction, changing from moving in the defocusing direction to moving in the focusing direction, until it reaches the camera module position corresponding to the target focus.

[0068] In some embodiments, the above method further comprises:

[0069] Based on the brightness change value of each pixel in the dynamic visual event obtained during the focusing process, the light spot brightness information of each light spot in the image frame is generated. The light spot brightness information includes the brightness change value of the target pixel associated with the light spot. The target pixel is the pixel that undergoes brightness change, and the target pixel is located at least one of the following: inside the light spot and at the edge of the light spot;

[0070] Based on the spherical aberration information of the camera module and the brightness information of each light spot, the foreground and background information of the light spot is determined. The spherical aberration information is used to indicate whether the spherical aberration is over-corrected, no spherical aberration, or under-corrected. The foreground and background information is used to indicate whether the light spot is a foreground light spot or a background light spot.

[0071] Based on the foreground and background information of each light spot, the corresponding light spot in the imaging image is blurred.

[0072] In this embodiment, the dynamic visual events obtained during the focusing process and the spherical aberration information of the camera module can be used to determine the foreground and background information of the light spot sensed by the image sensor, and based on the foreground and background information of each light spot, the light spot to be blurred corresponding to the light spot in the imaging image is blurred, thereby improving the blurring effect of the light spot and further improving the quality of the captured imaging image.

[0073] The dynamic visual events acquired during the focusing process can be dynamic visual events acquired over any duration during the focusing process. For example, the electronic device can control a camera motor to move the lens module from defocus to focus and then back to defocus, and acquire the dynamic visual events E(x, y, P, t) during this process.

[0074] The above-mentioned method of generating the light spot brightness information of each light spot sensed by the image sensor based on the brightness change value of each pixel point in the dynamic visual event obtained during the focusing process can be to extract the final brightness change value of the target pixel point associated with each light spot in the dynamic visual event to generate the light spot brightness information of the light spot.

[0075] The above-mentioned blurring of the corresponding light spots in the imaged image based on the foreground and background information of each light spot can be performed by adding the shape and brightness information of each light spot to the imaged image to form the light spots to be blurred in the imaged image; and then blurring the light spots to be blurred in the imaged image based on the foreground and background information of the light spots to be blurred.

[0076] The shape of the light spot is related to the shape of the aperture. For example, when the shape of the aperture is polygonal, the shape of the light spot is polygonal; and when the shape of the aperture is circular, the shape of the light spot is circular, and so on.

[0077] Of course, the shape of the above-mentioned light spot can also be adjusted according to the shape input by the user. For example, when the shape of the above-mentioned light spot is circular, if the user inputs a heart shape, then during the blurring process, the electronic device can convert the circular light spot into a heart-shaped light spot for blurring, and so on.

[0078] The above-mentioned blurring of the light spot to be blurred in the imaging image based on the foreground and background information of the light spot to be blurred may be performed by using an algorithm for blurring the foreground light spot when the light spot corresponding to the light spot to be blurred (that is, the light spot to be blurred is generated by the shape of the light spot and the brightness information of the light spot) is a foreground light spot; conversely, when the light spot corresponding to the light spot to be blurred is a background light spot, the light spot to be blurred is blurred by using an algorithm for blurring the background light spot.

[0079] When the spherical aberration is overcorrected, the edge of the foreground spot is relatively soft, such as Figure 4 The light spot in the first row and first column shown in the figure; the edge brightness of the background light spot is higher than the center brightness, forming a sharp edge with high brightness, as shown in the figure. Figure 4 The light spot in the first row and third column is shown.

[0080] When the spherical aberration is under-corrected, the edge of the background spot is relatively soft, such as Figure 4 The light spot in the third row and third column shown in the figure; the edge brightness of the foreground light spot is higher than the center brightness, forming a sharp edge brightness, as shown in the figure. Figure 4 The light spot in the third row and first column is shown.

[0081] In some embodiments, determining whether a light spot is a foreground light spot or a background light spot based on the spherical aberration information of the camera module and the light spot brightness information of each light spot includes:

[0082] When the spherical aberration information indicates that the spherical aberration is overcorrected and the light spot brightness information indicates that the light spot edge transition is soft, the light spot is determined to be a foreground light spot;

[0083] When the spherical aberration information indicates that the spherical aberration is overcorrected, and the spot brightness information of the spot indicates that the edge brightness of the spot is higher than the center brightness and forms a sharp edge with high brightness, the spot is determined to be a background spot;

[0084] When the spherical aberration information indicates that the spherical aberration is under-corrected and the light spot brightness information indicates that the light spot edge transition is soft, the light spot is determined to be a background light spot; or

[0085] When the spherical aberration information indicates that the spherical aberration is undercorrected and the spot brightness information of the spot indicates that the edge brightness of the spot is higher than the center brightness and forms a sharp edge with high brightness, the spot is determined to be a foreground spot.

[0086] In this way, it is possible to accurately determine whether the light spot is a foreground light spot or a background light spot, thereby improving the blurring effect of the light spot and further improving the quality of the captured image.

[0087] For example, the lens aperture shape can be combined with the characteristic of the DVS sensor outputting a time when exceeding a threshold value to determine whether the characteristics of the foreground and background changes of the light spot in the same location area and different time events meet the requirements of the optical imaging system with spherical aberration, thereby determining whether the light spot is a foreground light spot or a background light spot. The specific details are as follows:

[0088] Case 1: Spherical aberration is negative (i.e., spherical aberration is overcorrected) and the edge of the light spot is soft:

[0089] Assume that the light spot is circular, with the center position at (x0, y0), and the diameter of the circle is r. The lens module is focused or defocused (the characteristics of the two moving light spots are the same, the only difference is the expansion or reduction of the light spot area). If the lens moves toward defocus, the light spot spreads outward, and the brightness spreads outward in the shape of an aperture. The brightness of the center of the light spot will decrease slightly, and the DVS output event polarity P is -1. The number of output events is small, recorded as Sum(x,y,t); and the area exceeding the circle diameter r (i.e. ), the brightness area increases rapidly, the DVS output event polarity P is 1, the output event range is wide, but the brightness change is small. In summary, the brightness change value of a single pixel can be expressed by the following formula (3):

[0090] L(x,y)=Threshold*Sum(x,y,t)*P(1) (3)

[0091] In formula (3), Threshold is the DVS threshold. At this point, simply obtaining the final result of the brightness change L of a single pixel within the spot yields the spot information. Using ↑↓ to describe increases and decreases, with more arrows indicating greater changes, the following formula (4) indicates that the spot is a foreground spot.

[0092]

[0093] In the upper formula of formula (4), it means that the brightness of the pixel decreases within the circular spot; in the lower formula, it means that the brightness of the pixel increases as the circular spot spreads outward (i.e., at the edge of the circular spot);

[0094] Case 2: Spherical aberration is positive (i.e., spherical aberration is undercorrected) and the edge of the light spot is sharp:

[0095] Assuming the lens is defocused, the brightness at the center of the spot decreases slightly. The DVS output event polarity P is -1, but the number of output events is small. The spot diffuses outward, and the brightness spreads outward in the shape of an aperture. The diffusion area is small, but the brightness changes greatly. In this case, the spot is a background spot, which can be expressed by the following formula (5).

[0096]

[0097] In some embodiments, before generating the spot brightness information of each light spot sensed by the image sensor based on the brightness change value of each pixel in the dynamic visual event obtained during the focusing process, the method further includes:

[0098] Get the image frames collected by the sensor;

[0099] A light spot is determined within the subject of the image frame.

[0100] In this embodiment, by identifying light spots in the photographed object of the image frame captured by the sensor, the quality of the captured image is improved.

[0101] The above-mentioned identification of light spots in the photographed objects sensed by the sensor based on the image in the image frame can be scene recognition and object recognition of the image in the image frame, which assists in judging whether each photographed object in the current picture is a luminous object or a reflective object, thereby determining whether the photographed object is a light spot, and performing the above-mentioned blurring processing if it is a light spot.

[0102] The shooting method provided in the embodiment of the present application can be executed by a shooting device. In the embodiment of the present application, the shooting method is executed by a shooting device as an example to illustrate the shooting device provided in the embodiment of the present application.

[0103] See Figure 5 , the embodiment of the present application provides a structural diagram of a shooting device, which is applied to an electronic device including a camera module. Figure 5 As shown, the apparatus 500 includes:

[0104] A dynamic visual event acquisition module 501 is used to acquire dynamic visual events generated by the image sensor of the camera module when capturing each image frame during the focusing process of the camera module. The dynamic visual events include brightness change information of at least some pixels.

[0105] A brightness change information determination module 502 is configured to determine a brightness change area of an image frame corresponding to a dynamic visual event based on brightness change information of pixels in each dynamic visual event;

[0106] A position determination module 503 is used to determine the camera module position corresponding to the target focus based on the brightness change area of multiple image frames;

[0107] The focusing module 504 is used to control the camera module to focus on the target focus based on the camera module position corresponding to the target focus.

[0108] In some embodiments, the location determination module 503 includes:

[0109] A fitting unit, used to fit the camera module position and the brightness change area of the image frame to obtain a target quadratic curve;

[0110] The position determination unit is used to determine the camera module position corresponding to the maximum value of the target quadratic curve as the camera module position corresponding to the target focus.

[0111] In some embodiments, the apparatus 500 further comprises:

[0112] The focus state determination module is used to determine the focus state of the camera module based on the change trend of the brightness change area of multiple image frames. The change trend includes whether the brightness change area becomes larger or smaller, and the focus state includes the focused state or the out-of-focus state.

[0113] The focus module can be used for:

[0114] Based on the focusing state and the camera module position corresponding to the target focus, the camera module is controlled to focus on the target focus.

[0115] In some embodiments, the apparatus 500 further comprises:

[0116] A light spot brightness information generation module is configured to generate light spot brightness information for each light spot in an image frame based on the brightness change value of each pixel in a dynamic visual event acquired during the focusing process. The light spot brightness information includes the brightness change value of a target pixel associated with the light spot. The target pixel is a pixel that experiences a brightness change and is located at least one of within the light spot and at the edge of the light spot.

[0117] A foreground and background information determination module is used to determine the foreground and background information of a light spot based on the spherical aberration information of the camera module and the brightness information of each light spot. The spherical aberration information is used to indicate whether the spherical aberration is over-corrected, no spherical aberration, or under-corrected. The foreground and background information is used to indicate whether the light spot is a foreground light spot or a background light spot.

[0118] The blur module is used to blur the corresponding light spots in the imaging image based on the foreground and background information of each light spot.

[0119] In some embodiments, the focusing process includes: the camera module moves from defocus to focus, and then moves from focus to defocus.

[0120] In some implementations, the foreground and background information determination module is specifically configured to:

[0121] When the spherical aberration information indicates that the spherical aberration is overcorrected and the light spot brightness information indicates that the light spot edge transition is soft, the light spot is determined to be a foreground light spot;

[0122] When the spherical aberration information indicates that the spherical aberration is overcorrected, and the spot brightness information of the spot indicates that the edge brightness of the spot is higher than the center brightness and forms a sharp edge with high brightness, the spot is determined to be a background spot;

[0123] When the spherical aberration information indicates that the spherical aberration is under-corrected and the light spot brightness information indicates that the light spot edge transition is soft, the light spot is determined to be a background light spot; or

[0124] When the spherical aberration information indicates that the spherical aberration is undercorrected and the spot brightness information of the spot indicates that the edge brightness of the spot is higher than the center brightness and forms a sharp edge with high brightness, the spot is determined to be a foreground spot.

[0125] In some embodiments, the apparatus 500 further comprises:

[0126] An image frame acquisition module, used to acquire image frames collected by the sensor;

[0127] The light spot recognition module is used to determine the light spot in the object of the image frame.

[0128] The shooting device in the embodiments of the present application can be an electronic device or a component of an electronic device, such as an integrated circuit or chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiments of the present application do not specifically limit this.

[0129] The shooting device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0130] The shooting device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented in the method embodiment achieve the same effect, and to avoid repetition, they will not be described here.

[0131] Alternatively, as Figure 6As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores a program or instruction that can be run on the processor 601, and when the program or instruction is executed by the processor 601, the various steps of the above-mentioned shooting device method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0132] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0133] Figure 7 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0134] The electronic device 700 includes but is not limited to components such as a radio frequency unit 701 , a network module 702 , an audio output unit 703 , an input unit 704 , a sensor 705 , a display unit 706 , a user input unit 707 , an interface unit 708 , a memory 709 , and a processor 710 .

[0135] Those skilled in the art will understand that the electronic device 700 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0136] The processor 710 is configured to:

[0137] During the focusing process of the camera module, dynamic visual events of the image sensor of the camera module when capturing each image frame are obtained, where the dynamic visual events include brightness change information of at least some pixels;

[0138] Determining the brightness change area of the image frame corresponding to the dynamic visual event based on the brightness change information of the pixel points in each dynamic visual event;

[0139] Determine the camera module position corresponding to the target focus based on the brightness change area of multiple image frames;

[0140] Based on the position of the camera module corresponding to the target focus, the camera module is controlled to focus on the target focus.

[0141] In some implementations, the processor 710 is specifically configured to:

[0142] Fit the camera module position and the brightness change area of the image frame to obtain the target quadratic curve;

[0143] The camera module position corresponding to the maximum value of the target quadratic curve is determined as the camera module position corresponding to the target focus.

[0144] In some implementations, the processor 710 is further configured to:

[0145] Determining the focus state of the camera module based on a change trend of the brightness change area of the multiple image frames, where the change trend includes whether the brightness change area increases or decreases, and the focus state includes a focused state or an out-of-focus state;

[0146] Based on the focusing state and the camera module position corresponding to the target focus, the camera module is controlled to focus on the target focus.

[0147] In some implementations, the processor 710 is further configured to:

[0148] Based on the brightness change value of each pixel in the dynamic visual event obtained during the focusing process, the light spot brightness information of each light spot in the image frame is generated. The light spot brightness information includes the brightness change value of the target pixel associated with the light spot. The target pixel is the pixel that undergoes brightness change, and the target pixel is located at least one of the following: inside the light spot and at the edge of the light spot;

[0149] Based on the spherical aberration information of the camera module and the brightness information of each light spot, the foreground and background information of the light spot is determined. The spherical aberration information is used to indicate whether the spherical aberration is over-corrected, no spherical aberration, or under-corrected. The foreground and background information is used to indicate whether the light spot is a foreground light spot or a background light spot.

[0150] Based on the foreground and background information of each light spot, the corresponding light spot in the imaging image is blurred.

[0151] In some embodiments, the focusing process includes: the camera module moves from defocus to focus, and then moves from focus to defocus.

[0152] In some implementations, the processor 710 is specifically configured to:

[0153] When the spherical aberration information indicates that the spherical aberration is overcorrected and the light spot brightness information indicates that the light spot edge transition is soft, the light spot is determined to be a foreground light spot;

[0154] When the spherical aberration information indicates that the spherical aberration is overcorrected, and the spot brightness information of the spot indicates that the edge brightness of the spot is higher than the center brightness and forms a sharp edge with high brightness, the spot is determined to be a background spot;

[0155] When the spherical aberration information indicates that the spherical aberration is under-corrected and the light spot brightness information indicates that the light spot edge transition is soft, the light spot is determined to be a background light spot; or

[0156] When the spherical aberration information indicates that the spherical aberration is undercorrected and the spot brightness information of the spot indicates that the edge brightness of the spot is higher than the center brightness and forms a sharp edge with high brightness, the spot is determined to be a foreground spot.

[0157] In some implementations, the processor 710 is further configured to:

[0158] Get the image frames collected by the sensor;

[0159] A light spot is determined within the subject of the image frame.

[0160] The electronic device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented in the method embodiment achieve the same effect, and to avoid repetition, they will not be described here.

[0161] It should be understood that in an embodiment of the present application, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0162] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0163] Processor 710 may include one or more processing units. Optionally, processor 710 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 710.

[0164] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned shooting method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0165] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0166] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned shooting method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0167] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0168] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned shooting method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0169] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0171] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A shooting method, applied to an electronic device including a camera module, characterized in that: The method comprises: During the focusing process of the camera module, obtaining dynamic visual events of the image sensor of the camera module when capturing each image frame, wherein the dynamic visual events include brightness change information of pixel points; Determining a brightness change area of an image frame corresponding to the dynamic visual event based on brightness change information of pixels in each of the dynamic visual events; Determining a camera module position corresponding to a target focus according to brightness change areas of the plurality of image frames; Based on the position of the camera module corresponding to the target focus, controlling the camera module to focus on the target focus; generating, based on a brightness change value of each pixel in a dynamic visual event acquired during a focusing process, light spot brightness information for each light spot in the image frame, the light spot brightness information including a brightness change value of a target pixel associated with the light spot, the target pixel being a pixel at which the brightness change occurs, and the target pixel being located at least one of within the light spot and at an edge of the light spot; Determining foreground and background information of the light spot based on spherical aberration information of the camera module and light spot brightness information of each light spot, wherein the spherical aberration information is used to indicate whether the spherical aberration is overcorrected, no spherical aberration, or undercorrected, and the foreground and background information is used to indicate whether the light spot is a foreground light spot or a background light spot; Based on the foreground and background information of each light spot, the corresponding light spot in the imaging image is blurred.

2. The method according to claim 1, characterized in that Determining the camera module position corresponding to the target focus according to the brightness change area of the plurality of image frames includes: Fit the camera module position and the brightness change area of the image frame to obtain the target quadratic curve; The camera module position corresponding to the maximum value of the target quadratic curve is determined as the camera module position corresponding to the target focus.

3. The method according to claim 1, characterized in that After determining the brightness change area of the image frame corresponding to the dynamic visual event based on the brightness change information of the pixel points in each dynamic visual event, the method further includes: Determining a focus state of the camera module based on a change trend of the brightness change area of the plurality of image frames, wherein the change trend includes an increase or decrease in the brightness change area, and the focus state includes a focused state or a defocused state; The controlling the camera module to focus on the target focus based on the camera module position corresponding to the target focus includes: Based on the focusing state and the camera module position corresponding to the target focus, the camera module is controlled to focus on the target focus.

4. The method according to claim 1, wherein The focusing process includes: the camera module moves from defocus to focus, and then moves from focus to defocus.

5. The method according to claim 1, wherein The determining whether the light spot is a foreground light spot or a background light spot based on the spherical aberration information of the camera module and the light spot brightness information of each light spot includes: When the spherical aberration information indicates that the spherical aberration is overcorrected and the light spot brightness information indicates that the light spot edge transition is soft, determining that the light spot is a foreground light spot; When the spherical aberration information indicates that the spherical aberration is overcorrected, and the spot brightness information of the light spot indicates that the edge brightness of the light spot is higher than the center brightness and forms a sharp edge with high brightness, determining that the light spot is a background light spot; When the spherical aberration information indicates that the spherical aberration is under-corrected and the light spot brightness information of the light spot indicates that the light spot edge transition is soft, determining that the light spot is a background light spot; or When the spherical aberration information indicates that the spherical aberration is undercorrected and the spot brightness information of the light spot indicates that the edge brightness of the light spot is higher than the center brightness and forms a sharp edge with high brightness, the light spot is determined to be a foreground spot.

6. The method according to claim 1, characterized in that Before generating the spot brightness information of each light spot sensed by the image sensor based on the brightness change value of each pixel point in the dynamic visual event obtained during the focusing process, the method further includes: Acquire image frames captured by the sensor; The light spot is determined in the photographed object of the image frame.

7. A shooting device, applied to an electronic device including a camera module, characterized in that: The device comprises: A dynamic visual event acquisition module, configured to acquire dynamic visual events generated by the image sensor of the camera module when capturing each image frame during the focusing process of the camera module, wherein the dynamic visual events include brightness change information of at least some pixels; a brightness change information determining module, configured to determine a brightness change area of an image frame corresponding to a dynamic visual event based on brightness change information of pixels in each dynamic visual event; A position determination module, configured to determine a camera module position corresponding to a target focus based on brightness change areas of a plurality of image frames; A focusing module, configured to control the camera module to focus on the target focus based on the camera module position corresponding to the target focus; a light spot brightness information generation module, configured to generate light spot brightness information for each light spot in the image frame based on the brightness change value of each pixel in the dynamic visual event acquired during the focusing process, wherein the light spot brightness information includes the brightness change value of a target pixel associated with the light spot, the target pixel being a pixel at which the brightness change occurs, and the target pixel being located at least one of within the light spot and at the edge of the light spot; a foreground and background information determination module, configured to determine foreground and background information of the light spot based on spherical aberration information of the camera module and spot brightness information of each light spot, wherein the spherical aberration information is used to indicate whether the spherical aberration is overcorrected, no spherical aberration, or undercorrected, and the foreground and background information is used to indicate whether the light spot is a foreground light spot or a background light spot; The blurring module is used to blur the corresponding light spot in the imaging image based on the foreground and background information of each light spot.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the shooting method according to any one of claims 1 to 6 are implemented.

9. A readable storage medium, characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the shooting method according to any one of claims 1 to 6 are implemented.

Images