Focus tracking method and electronic equipment

Abstract

The invention provides a focus tracking method and electronic equipment. By implementing the focus tracking method provided by the invention, the electronic equipment, such as a mobile phone, a tablet computer and the like, which is provided with a camera and has a shooting capability, can identify one or more specific objects in the image after acquiring the latest frame of image; the method comprises the following steps: acquiring one or more specific objects, determining a preferable object which has the highest overlapping degree with a shooting focus of a previous frame of image from the one or more specific objects, then determining a target object according to the shooting focus in the previous frame of image and the preferable object in a latest frame of image, and determining the target object as the shooting focus of the latest frame of image. In this way, the change of the shooting focus between the front and back two frames of images is smoother, and the jumping of the image content between clearness and blurring caused by the jump of the shooting focus can be avoided. And the user can obtain smoother and more comfortable automatic focus tracking shooting experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the terminal field, and in particular to a focus tracking method and an electronic device. BACKGROUND

[0002] Currently, electronic devices with shooting capabilities usually also provide an automatic focusing function. When the automatic focusing function is turned on, the electronic device can identify a specific object in an image captured by a camera, such as a person, an animal, a plant, a building, and the like, and then the electronic device can adjust the focal length of the camera according to the depth of field of the specific object identified, so that the specific object is imaged clearly. In one focus tracking method, the electronic device determines, by default, the object with the highest confidence among one or more objects identified as the shooting focus. At this time, the electronic device is prone to the problem of frequent changes in the shooting focus, which affects the user's shooting experience. SUMMARY

[0003] The present application provides a focus tracking method and an electronic device.

[0004] In a first aspect, the present application provides a focus tracking method applied to an electronic device, the method comprising: obtaining a first image captured by a camera, and determining a first region as a focusing region of the first image; obtaining a second image captured by the camera, and when the second image includes N2 candidate focusing regions corresponding to N2 objects, determining a first candidate focusing region from the N2 candidate focusing regions, N2≥1, and the intersection over union (IoU) max of the first candidate focusing region and the first region is the largest; determining a second region according to the first region and the first candidate focusing region, and determining the second region as a focusing region of the second image.

[0005] By implementing the above method, after obtaining the latest image, the electronic device can identify one or more specific objects in the image, determine one or more candidate focusing regions corresponding to the one or more specific objects, and then determine a candidate focusing region (i.e., a first focusing region) with the highest overlap with the shooting focus of the previous image (i.e., the focusing region of the previous image) from the one or more candidate focusing regions, and then determine a focusing region (a second region) of the second image according to the focusing region of the previous image and the first focusing region.

[0006] In this way, the change of the shooting focus between the two images is smoother, and the jumping of the image content between clear and blurred caused by the jumping of the shooting focus can be avoided. The user can obtain a smoother and more comfortable automatic focus tracking shooting experience.

[0007] In some embodiments, the second region is determined according to the first region and the first candidate focusing region, specifically comprising: determining the second region according to the first region, the first candidate focusing region, and the IoU max.

[0008] In some embodiments, the second region is determined according to the first region, the first candidate focusing region and the IoUmax, specifically comprising: St=Sx*(1-IoUmax)+S1*IoUmax; wherein S1 represents the coordinates of the first region, Sx represents the coordinates of the first candidate focusing region, and St represents the coordinates of the second region.

[0009] In some embodiments, the second region is determined according to the first region, the first candidate focusing region and the IoUmax, specifically comprising: St=Sx*(1-IoUmax)+S1*IoUmax; wherein S1 represents the coordinates of the first region, Sx represents the coordinates of the first candidate focusing region, and St represents the coordinates of the second region.

[0010] The change between the second region determined by the above method and the first region is more smooth, which is beneficial to avoid the jumping of the shooting focus, and further avoid the jumping of the image content between clear and blurred, so that the user can obtain a more smooth and comfortable automatic focus shooting experience.

[0011] In some embodiments, the second region is determined according to the first region and the first candidate focusing region, specifically comprising: when the IoUmax is greater than a first threshold M1, the second region is determined according to the first region and the first candidate focusing region.

[0012] In some embodiments, the above method further comprises: when the IoUmax is less than or equal to M1, increasing the count value C of the first counter; when C is greater than a second threshold M2, determining the first candidate focusing region as the focusing region of the second image; when C is less than or equal to M2, determining the second region according to the first region and the first candidate focusing region, and determining the second region as the focusing region of the second image.

[0013] In some embodiments, the above method further comprises: when the IoUmax is less than or equal to M1, increasing the count value C of the first counter; when C is greater than a second threshold M2, determining the first candidate focusing region as the focusing region of the second image; when C is less than or equal to M2, determining the first region as the focusing region of the second image.

[0014] In some embodiments, the above method further comprises: when the IoUmax is less than or equal to M1, determining the first candidate focusing region as the focusing region.

[0015] By implementing the above method, the electronic device can adopt different specific methods to determine the shooting focus in the new image according to the relationship between the IoUmax and the preset threshold M1, so as to realize different focus effects on the premise of avoiding frequent jumping of the shooting focus.

[0016] In some embodiments, the method further includes: displaying a shooting preview interface; displaying the first image in the shooting preview interface, the first image having the first mark displayed on the first region; and displaying the second image in the shooting preview interface, the second image having the second mark displayed on the second region.

[0017] In this way, the user can determine the current shooting focus in real time, so as to adjust at any time and meet the changing shooting needs of the user.

[0018] In some embodiments, the method further includes: when the second image does not have a candidate focusing region, determining the first region as the focusing region of the second image.

[0019] In a second aspect, the present application provides an electronic device, comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are configured to store a computer program, and when the one or more processors execute the computer program, the electronic device is caused to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0020] In a third aspect, the present application provides a chip system, applied to an electronic device, comprising one or more processors, and the processor is configured to invoke a computer instruction to cause the electronic device to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0021] In a fourth aspect, the present application provides a computer readable storage medium, comprising a computer program, and when the computer program runs on an electronic device, the electronic device is caused to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a fifth aspect, the present application provides a computer program product comprising instructions, and when the computer program product runs on an electronic device, the electronic device is caused to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0023] It can be understood that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in the present application. Therefore, the beneficial effects that can be achieved are referred to the beneficial effects in the corresponding method, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figures 1A-1B is a set of user interface diagrams of shooting focus jump provided by the embodiments of the present application;

[0025] Figure 2is a flowchart of a method for tracking focus provided by an embodiment of the present application;

[0026] Figure 3 is a schematic diagram for determining region IoU according to region coordinates provided by an embodiment of the present application;

[0027] Figure 4 is a schematic diagram of a target object provided by an embodiment of the present application;

[0028] Figure 5 is a flowchart of another method for tracking focus provided by an embodiment of the present application;

[0029] Figures 6A-6B is a set of tracking focus schematic diagrams provided by an embodiment of the present application;

[0030] Figure 7 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting to the present application.

[0032] Currently, electronic devices with shooting capabilities usually also provide an automatic focusing function. When the automatic focusing function is turned on, the electronic device can identify a specific object in the image captured by the camera, such as a person, an animal, a plant, a building, etc., and then the electronic device can adjust the focal length of the camera according to the depth of field of the identified specific object, so that the above-mentioned specific object is imaged clearly. At this time, the above-mentioned specific object is the shooting focus. The area where the shooting focus is located is the focus area. In a video stream, the action of determining the shooting focus of each frame of image is called tracking focus.

[0033] Usually, a frame of image can include multiple specific objects, such as multiple persons, persons and animals, persons and buildings, etc. At this time, the electronic device needs to select one from the multiple objects as the shooting focus. In one tracking focus method, the electronic device defaults to determine the object with the highest confidence among the identified one or more objects as the shooting focus. At this time, the electronic device is prone to the problem of frequent jumping of the shooting focus, which affects the user shooting experience.

[0034] Figures 1A-1B is a set of user interface schematic diagrams of shooting focus jumping provided by an embodiment of the present application.

[0035] Reference Figure 1AAfter enabling autofocus, the electronic device can use an object recognition algorithm to identify objects 11, 12, and 13 in image 10 captured by the camera. Assume the confidence levels of objects 11, 12, and 13 output by the object recognition algorithm are C1, C2, and C3, respectively, where C1 > C2 > C3. Based on the confidence levels from highest to lowest, the electronic device can determine object 11, with the highest confidence level, as the focus point. Therefore, the electronic device can adjust the camera's focal length according to the depth of field of object 11, making object 11 appear sharp. At this point, objects 12 and 13, which are not within the same depth of field as object 11, will appear blurry.

[0036] Optionally, after determining the shooting focus, the electronic device may display a marker box on the shooting focus to indicate to the user that the object in the marker box is the current shooting focus. For example... Figure 1A As shown, after determining that the object 11 is the focus of the shot, the electronic device can display a marker box 101 on the object 11 to indicate to the user that the current focus of the shot is the object 11. The area covered by the marker box 101 is also the focus area of ​​the image 10, and the marker box 101 is also used to indicate the focus area of ​​the image 10.

[0037] refer to Figure 1B Following image 10, the camera can capture image 20. Similarly, based on the object recognition algorithm, the electronic device can identify objects 21, 22, and 23 in image 20, with confidence levels of C4, C5, and C6 respectively, where C5 > C4 > C6. At this point, the electronic device will determine object 22, with the highest confidence level, as the new focus point and adjust the camera's focal length according to the depth of field of object 22, making object 22 appear sharp. Objects 21 and 23, which are not within the same depth of field as object 22, will appear blurry. Similarly, after determining object 22 as the new focus point, the electronic device can display a marker box 201 on object 22 to indicate to the user that the current focus is on object 22. The area covered by the marker box 201 is also the focus area of ​​image 20, and the marker box 201 is used to indicate the focus area of ​​image 20.

[0038] like Figures 1A-1B As shown, during the process of switching the shooting focus from object 11 in the lower left region of the image to object 22 in the upper right region of the image, the sharp image area where the shooting focus is located undergoes a significant jump.

[0039] Similarly, the focus tracking method described above, which determines the shooting focus solely based on the level of confidence, does not involve the content relationship between the preceding and following images. This can easily lead to frequent changes in the shooting focus, causing the image content at the same location to be sometimes clear and sometimes blurry, thus affecting the user's shooting experience.

[0040] In view of this, the embodiment of the present application provides a method for tracking focus. The method can be applied to electronic devices such as mobile phones and tablet computers that are equipped with cameras and have the ability to take pictures. After the latest frame of image is obtained, the electronic device can identify one or more specific objects in the image, determine a preferred object with the highest overlap degree with the focus point of the previous frame of image from the one or more specific objects, and then determine a target object according to the focus point of the previous frame of image and the preferred object in the latest frame of image, and determine the target object as the focus point of the latest frame of image.

[0041] In this way, the change of the focus point between the front and back frames of images is smoother, and the jumping of the image content between clear and blurred can be avoided. Users can obtain a more smooth and comfortable automatic tracking focus shooting experience.

[0042] The electronic device implementing the above-mentioned tracking focus method can also be a sports camera, a aerial vehicle, an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device, etc. The type of the electronic device is not specially limited in the embodiment of the present application.

[0043] Figure 2 is a flowchart of the tracking focus method provided by the embodiment of the present application.

[0044] S211, obtain a first frame of image F1.

[0045] In some embodiments, after the camera is turned on, the electronic device turns on the automatic focusing function by default. At this time, the first frame of image collected by the camera is the first frame of image F1.

[0046] In some embodiments, after the camera is turned on, the electronic device turns off the automatic focusing function by default. After detecting a user operation of turning on the automatic focusing function, the electronic device turns on the automatic focusing function. At this time, the first frame of image collected by the camera after turning on the automatic focusing function is the first frame of image F1.

[0047] S212, determine N1 objects in F1, N1≥1?

[0048] The electronic device can process F1 through a preset object recognition algorithm to obtain specific objects in F1.

[0049] When F1 includes at least one (i.e., includes N1, N1≥1) preset specific object (e.g., a person, an animal, a plant, a building, etc.), the electronic device immediately performs S213. Conversely, if no specific object is identified from F1, the electronic device acquires the next frame image captured by the camera, again performs S212, until at least one preset specific object is identified, and then performs S213.

[0050] S213, determines the object with the highest confidence among the N1 objects as the shooting focus, and adjusts the focal length of the camera according to the depth of field of the shooting focus, so that the shooting focus is imaged clearly.

[0051] S214, acquires the next frame image F2.

[0052] S215, determines N2 objects in F2, N2≥1?

[0053] Similarly, when F2 includes at least one (i.e., includes N2, N2≥1) preset specific object, the electronic device immediately performs S216. Conversely, if no specific object is identified in F2, the electronic device can use the coordinates of the shooting focus in the previous frame image, determine the image content at the corresponding position in the current frame image F2 as the shooting focus, then adjust the focal length of the camera according to the depth of field of the new shooting focus, so that the shooting focus in the current frame image F2 is imaged clearly, and continue to acquire the next frame image, again perform S215, until at least one preset specific object is identified, and then perform S216.

[0054] S216, determines the preferred object among the N2 objects that has the highest overlap with the shooting focus in the previous frame image.

[0055] The electronic device can represent the overlap of two regions by the Intersection of Union (IoU) of the two regions. Wherein, the higher the IoU of the two regions, the higher the overlap of the two regions; conversely, the lower the IoU of the two regions, the lower the overlap of the two regions. 0≤IoU≤1.

[0056] The electronic device can acquire the coordinates of the shooting focus in the previous frame image, and the coordinates of each object in the current frame image, and determine the IoU, i.e., the overlap, of each object in the current frame image with the shooting focus in the previous frame image according to the above coordinates.

[0057] Figure 3 is a schematic diagram provided by an embodiment of the present application for determining the IoU of a region according to region coordinates. As shown in FIG. 13, the IoU of a region A with a region B is determined according to the coordinates of the region A and the coordinates of the region B. Figure 3As shown, the dashed box 31 represents the region where the focus point is in the previous frame image, and the solid box 32 represents the region where an object is in the current frame image. For any rectangle, the position of the rectangle can be represented by the coordinates of the vertices on the diagonal. For example, the dashed box 31 can be represented by the coordinates S1 (X1, Y1, X2, Y2), and the solid box 32 can be represented by the coordinates S2 (X3, Y3, X4, Y4). Wherein (X1, Y1), (X2, Y2) are a set of diagonal vertices of the dashed box 31, and (X3, Y3), (X4, Y4) are a set of diagonal vertices of the solid box 32.

[0058] Therefore, the intersection I2 of the dashed box 31 and the solid box 32 can be represented as:

[0059] I2 = (X2-X3)(Y2-Y3);

[0060] The union U2 of the dashed box 31 and the solid box 32 can be represented as:

[0061] U2 = [(X4-X1)(Y4-Y1)-(X3-X1)(Y4-Y2)-(X4-X2)(Y3-Y1)];

[0062] The IoU2 of the dashed box 31 and the solid box 32 can be represented as:

[0063] IoU2 = (X2-X3)(Y2-Y3) / [(X4-X1)(Y4-Y1)-(X3-X1)(Y4-Y2)-(X4-X2)(Y3-Y1)].

[0064] In some embodiments, after determining the preferred object in the N2 objects that has the highest overlap with the focus point in the previous frame image, the electronic device can determine the preferred object as the focus point of the current frame image, and then adjust the focal length of the camera according to the depth of field of the preferred object to make the preferred object clear in the image.

[0065] The preferred object that has the highest overlap with the focus point in the previous frame image is also the object that is closest to the focus point in the previous frame image. In a short period of time, the position of the shooting object in the viewfinder range generally does not jump, and therefore, the object that is closest to the focus point in the previous frame image is also the most likely object that is the same as the focus point in the previous frame image. In this way, by implementing the above focus tracking method, the electronic device can automatically and continuously focus on the same object and perform a shooting operation to obtain a stable and smooth video, thereby avoiding the jumping of the focus point and the local image jump caused by the jumping of the focus point.

[0066] Preferably, in this embodiment of the application, after determining the preferred object with the highest overlap with the shooting focus in the previous frame image among the N2 objects, that is, after executing S216, the electronic device can execute S217 to further avoid shooting focus jump.

[0067] S217. Determine the target object based on the shooting focus of the previous frame and the preferred object of the current frame, and set the target object as the shooting focus of the current frame.

[0068] In one embodiment, the electronic device can determine the coordinates St of the target object according to the following formula (1).

[0069] St=Sx*(1-IoUmax)+S1*IoUmax;

[0070] Where Sx represents the coordinates of the preferred object in the current frame image, such as S2(X3,Y3,X4,Y4) shown in S216, S1 represents the coordinates of the shooting focus in the previous frame image, such as S1(X1,Y1,X2,Y2), and IoUmax represents the intersection-over-union ratio of the preferred object and the shooting focus in the previous frame.

[0071] Assuming, Figure 3 The object shown in the solid box 32 is the preferred object with the highest overlap with the focus of the previous frame image, so IoUmax = IoU2. Substituting S1(X1,Y1,X2,Y2), S2(X3,Y3,X4,Y4) and IoU2 into formula (1) yields the coordinates St of the target object:

[0072] St=(X3,Y3,X4,Y4)*(1-IoU2)+(X1,Y1,X2,Y2)*IoU2;

[0073] = (X5, Y5, X6, Y6);

[0074] Therefore, the electronic device can determine that the image content corresponding to St(X5,Y5,X6,Y6) in the current frame image is the target object.

[0075] Figure 4 This is a schematic diagram of the target object provided in the embodiments of this application. For example... Figure 4 As shown, the image content enclosed by the solid-line box 33 corresponding to St(X5,Y5,X6,Y6) in the current frame image is the target object. Therefore, the electronic device can adjust the camera focal length according to the depth of field of the target object to make the target object clear.

[0076] After performing S217, the electronic device can acquire a next frame image, for example, a third frame image F3 captured by the camera, perform the operations shown in S214-S217 on F3, determine the highest overlap degree preferred object in F3 with the shooting focus of the previous frame image F2, and further determine the target object of F3, and determine the shooting focus of F3. Here, no longer described.

[0077] Compared with the method of directly setting the preferred object as the shooting focus as shown in Figure 3 the method of determining the target object according to the shooting focus of the previous frame and the preferred object of the current frame, and then setting the target object as the shooting focus as shown in Figure 4 the method of determining the target object according to the shooting focus of the previous frame and the preferred object of the current frame, and then setting the target object as the shooting focus as shown in

[0078] In some embodiments, the electronic device can also determine the coordinates St of the target object according to the following formula (2),

[0079] St=Sx*IoUmax+Si*(1-IoUmax);

[0080] According to formula (2), the electronic device can further constrain the change range of the shooting focus, avoid the shooting focus jump, and make the user obtain more stable and smoother video.

[0081] Figure 5 is a flowchart of another focus tracking method provided by the embodiments of the present application.

[0082] As shown in Figure 5 After S216 and before S217, preferably, the electronic device can further determine whether the intersection-over-union IoU-max of the preferred object and the shooting focus of the previous frame is greater than a threshold M1. When IoU-max>M1, the electronic device immediately performs S217. IoU-max≤M1 means that the preferred object and the shooting focus of the previous frame do not overlap or the overlapping area is too small, that is, the distance between the preferred object and the shooting focus of the previous frame is not close enough.

[0083] In some embodiments, when IoU-max≤M1, the electronic device can use the coordinates of the shooting focus in the previous frame image, determine the image content at the corresponding position in the current frame image as the shooting focus, then adjust the focal length of the camera according to the depth of field of the new shooting focus, make the shooting focus in the current frame image clear, and then the electronic device continues to acquire the next frame image and executes S215 and the processes after S215 again.

[0084] In some embodiments, when IoU-max≤M1, the electronic device can also directly set the preferred object as the shooting focus of the current frame.

[0085] Preferably, in other embodiments, as shown, when IoU-max≤M1, the electronic device can increase the count value of the counter. Then, the electronic device can determine whether the count value coun of the counter is greater than a threshold M2. coun>M2 means that, for a long period of time, the preferred object in each frame of image is far away from the previously determined shooting focus, and the previous shooting focus no longer meets the shooting needs of the user. Therefore, when coun≤M2, the electronic device can perform S217 to determine the target object according to the shooting focus position of the previous frame of image and the preferred object of the current frame, and set the target object as the shooting focus of the current frame. When coun>M2, the electronic device can directly set the preferred object as the shooting focus of the current frame. Figure 5 Alternatively, in some embodiments, when coun≤M2, the electronic device can also use the coordinates of the shooting focus of the previous frame of image, and determine the image content at the corresponding position in the current frame of image as the shooting focus. When coun>M2, the electronic device can directly set the preferred object as the shooting focus of the current frame.

[0086]

[0087] Figures 6A-6B is a set of focus-pursuing schematic diagrams provided by an embodiment of the present application.

[0088] After the camera is turned on, the camera can periodically generate a frame of image according to the collected light signals. For example, taking a frame rate of 30 FPS as an example, the camera can generate a frame of image every 33.3 ms. The images generated continuously by the camera are called video stream.

[0089] In the default scenario of turning on the automatic focusing function, after obtaining the first frame of image F1 generated by the camera, the electronic device can identify the objects in F1. Referring to Figure 6A , for example, the electronic device can identify that F1 includes objects T1, T2 and T3, and determine the confidence of the above-mentioned objects T1, T2 and T3. For example, the confidence of T1 is 0.96, the confidence of T2 is 0.90, and the confidence of T3 is 0.87. At this time, the electronic device can determine that the object with the highest confidence in F1 is T1. Then, the electronic device can determine T1 as the shooting focus, and then adjust the focal length of the camera according to the depth of field of T1, so that the shooting focus T1 is imaged clearly. At this time, the objects T2 and T3 which are not in the same depth of field range as the shooting focus T1 are imaged blurredly.

[0090] ​Subsequently, the electronic device acquires the second frame image F2 generated by the camera. Similarly, the electronic device first identifies the objects in F2. For example, the electronic device can identify objects T5, T6, and T7 in F2, and simultaneously determine the confidence level of each object. For example, the confidence level of T5 is 0.95, the confidence level of T6 is 0.96, and the confidence level of T7 is 0.87. At this point, if the conventional method of determining the shooting focus based on confidence level is used, the electronic device would determine the object T6 with the highest confidence level as the shooting focus.

[0091] However, implementation Figure 2 The method shown allows the electronic device to determine the overlap between objects T5, T6, and T7 and the focus point in the previous frame image after identifying objects T5, T6, and T7; that is, the Intersection over Union (IoU) between objects T5, T6, and T7 and the focus point in the previous frame image. The IoU ratio of T5 to the focus point in the previous frame image is denoted as IoU5, T6 as IoU6, and T7 as IoU7. For example... Figure 6A As shown, the dashed box 601 in F2 represents the focus point T1 in the previous frame. Obviously, the electronic device can determine that IOU5 > IOU6 > IOU7, and thus determine that T5 is the object with the highest overlap with the focus point in the previous frame.

[0092] Therefore, the electronic device can determine T5 as the preferred object. Then, the electronic device can determine the target object (the image content shown in the dashed box 602) based on the shooting focus T1 in the previous frame image F1 and the preferred object T5, set the target object as the shooting focus of F2, and then adjust the camera focal length according to the depth of field of the shooting focus to make the image of the shooting focus clear. The above process can be referred to in detail in the description of S217, and will not be repeated here.

[0093] refer to Figure 6B After acquiring the third frame image F3 generated by the camera, the electronic device first identifies the objects in F3. For example, the electronic device may identify objects T8 and T9 in F3 and simultaneously determine the confidence levels of these objects, such as 0.94 and 0.93. Then, the electronic device may identify the preferred object from among objects T8 and T9 that has the highest overlap with the focus of the previous frame image F2, for example, T8. Then, the electronic device may determine the target object (the image content shown in dashed box 603) based on the focus of the previous frame image F2 (i.e., the image content shown in dashed box 602) and the preferred object T8, set the target object as the focus of F3, and then adjust the camera focal length according to the depth of field of the focus to ensure a clear image of the focus.

[0094] By analogy, the electronic device can constrain the range of the change of the shooting focus, avoid the jumping of the shooting focus, and thus avoid the local image jumping caused by the jumping of the shooting focus, so that the user obtains more stable and smoother video, and the use experience of the user is improved.

[0095] In Figures 6A-6B In the example shown in the figure, the second frame image F2 can be referred to as a first image, and the area surrounded by the dashed box 602 can be referred to as a first area; the third frame image F3 can be referred to as a second image, and the areas where the objects T8 and T9 are located in F3 are two candidate focusing areas. Among them, the area where T8, which has the highest overlap degree with the shooting focus (that is, the first area) in the previous frame image F2, is located is a first candidate focusing area, and the area surrounded by the dashed box 603 is a second area.

[0096] With reference to Figures 1A-1B For the shooting focus, the electronic device can display a mark box on the shooting focus. Therefore, in the shooting preview interface, when F2 is displayed, the electronic device can display a mark box (a first mark) at the position of the dashed box 602 to indicate that the image content in the dashed box 602 is the shooting focus, that is, to indicate that the image content in the dashed box 602 is the focusing area; similarly, when F3 is displayed, the electronic device can display a mark box (a second mark) at the position of the dashed box 603 to indicate that the image content in the dashed box 603 is the shooting focus.

[0097] Figure 7 FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0098] The electronic device can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0099] It can be understood that the structure shown in the embodiment of the present application does not constitute a specific limitation on the electronic device. In another embodiment of the present application, the electronic device can include more or fewer components than the diagram, or combine certain components, or split certain components, or different component arrangements. The components shown in the diagram can be implemented in hardware, software, or a combination of software and hardware.

[0100] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0101] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instructions or data again, it can directly call them from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0102] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, etc.

[0103] For example, the processor 110 can be coupled with the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate with each other through an I2C bus interface, to realize the touch function of the electronic device; the processor 110 can be coupled with the audio module 170 through an I2S bus, to realize the communication between the processor 110 and the audio module 170; the audio module 170 can deliver an audio signal to the wireless communication module 160 through an I2S interface, to realize the function of answering a phone call through a Bluetooth headset; the audio module 170 can also deliver an audio signal to the wireless communication module 160 through a PCM interface, to realize the function of answering a phone call through a Bluetooth headset; the audio module 170 can also deliver an audio signal to the wireless communication module 160 through a UART interface, to realize the function of playing music through a Bluetooth headset. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate with each other through a CSI interface, to realize the shooting function of the electronic device. The processor 110 and the display 194 communicate with each other through a DSI interface, to realize the display function of the electronic device. The USB interface 130 can be used to connect a charger to charge the electronic device, or to transmit data between the electronic device and a peripheral device. It can also be used to connect a headset to play audio through the headset.

[0104] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative, and does not constitute a structural limitation on the electronic device. In some other embodiments of the present application, the electronic device can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0105] The charging management module 140 is configured to receive a charging input from a charger. The charging management module 140 can also supply power to the electronic device through the power management module 141 while charging the battery 142. The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives the input of the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160, etc.

[0106] The wireless communication function of the electronic device can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, and a baseband processor, etc.

[0107] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device. The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the electronic device. In some embodiments, the antenna 1 and the mobile communication module 150 are coupled, and the antenna 2 and the wireless communication module 160 are coupled, so that the electronic device can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0108] The electronic device implements a display function through a GPU, a display screen 194, an application processor, and the like, and displays Figures 1A-1B The electronic device displays a preview interface as shown.

[0109] The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information. The display screen 194 is used to display images, videos, and the like. The display screen 194 includes a display panel. The display panel can be manufactured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniled, a microled, a micro-oled, a quantum dot light emitting diode (QLED), and the like. In some embodiments, the electronic device can include 1 or N display screens 194, N being a positive integer greater than 1.

[0110] The electronic device can implement a photographing function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, and the like.

[0111] The camera 193 is used to capture still images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP is used to process the data fed back by the camera 193 and convert it into an image visible to the naked eye. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, the electronic device can include 1 or N cameras 193, N being a positive integer greater than 1.

[0112] A video codec is used to compress or decompress digital video. An electronic device can support one or more video codecs. In this way, the electronic device can play or record videos in a variety of encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and so on.

[0113] The internal memory 121 can include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs).

[0114] The RAM can include a static random-access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDR SDRAM, such as a fifth generation DDR SDRAM commonly referred to as a DDR5 SDRAM), and so on. The NVM can include a disk storage device, a flash memory.

[0115] The RAM can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of an operating system or other programs that are running, and can also be used to store data of users and application programs, and so on. The NVM can also store executable programs and store data of users and application programs, and can be loaded into the RAM in advance for direct reading and writing by the processor 110.

[0116] In the embodiments of the present application, executable program codes for implementing the focus-pursuing method described in the embodiments of the present application can be stored in the NVM. After the electronic device is powered on and started, the electronic device can load the above-mentioned executable program codes stored in the NVM into the RAM, so that the electronic device can automatically and continuously focus on the same object and perform a shooting operation to obtain a stable and smooth video, thereby avoiding a shooting focus jump and a local image jump caused by the shooting focus jump.

[0117] The external memory interface 120 can be used to connect an external NVM to expand the storage capacity of the electronic device. In some embodiments, the executable program code implementing the focus tracking method described in the embodiments of the present application can also be stored in the external NVM connected by the external memory interface 120.

[0118] The electronic device can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playing, recording, etc.

[0119] The keys 190 include a power key, a volume key, etc. The keys 190 can be mechanical keys. They can also be touch keys. The electronic device can receive key input through the keys 190 and generate key signal input related to user settings and function control of the electronic device. The motor 191 can generate a vibration prompt. The indicator 192 can be an indicator light, which can be used to indicate the charging state, the power change, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 is used to connect the SIM card.

[0120] The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0121] The pressure sensor 180A is configured to sense a pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. When a touch operation is performed on the display screen 194, the electronic device detects the intensity of the touch operation according to the pressure sensor 180A. The electronic device can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. The gyroscope sensor 180B can be configured to determine the angular velocity of the electronic device around three axes (i.e., x, y, and z axes). The acceleration sensor 180E can be configured to detect the magnitude of acceleration of the electronic device in each direction (typically three axes). The electronic device can determine the motion attitude of the electronic device through the gyroscope sensor 180B and the acceleration sensor 180E. In a shooting scenario, the gyroscope sensor 180B can also be used for shooting anti-shake. The barometric pressure sensor 180C is configured to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The electronic device can detect the opening and closing of a flip cover through the magnetic sensor 180D, and then set a flip automatic unlocking feature according to the detected opening and closing state of the cover or the opening and closing state of the flip. The distance sensor 180F is configured to measure distance. In a shooting scenario, the electronic device can determine the depth of field of a shooting object through the distance sensor 180F, and then adjust the focal length of the camera 193 according to the depth of field to achieve fast focusing. The proximity light sensor 180G can be used for object detection. The ambient light sensor 180L is configured to sense ambient light brightness. The electronic device can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust white balance when shooting. The fingerprint sensor 180H is configured to collect fingerprints. The temperature sensor 180J is configured to detect temperature. The bone conduction sensor 180M can obtain vibration signals.

[0122] The touch sensor 180K, also referred to as a "touch device". The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch screen". The touch sensor 180K is configured to detect a touch operation performed thereon or in the vicinity thereof. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. In turn, the electronic device can provide visual output related to the touch operation through the display screen 194.

[0123] The term "user interface (UI)" in the specification and claims of the present application and the accompanying drawings is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java, extensible markup language (XML), etc. The interface source code is parsed, rendered on a terminal device, and finally presented as content recognizable by the user, such as pictures, text, buttons, etc. controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scrollbars, pictures, and text. The properties and content of the controls in the interface are defined by tags or nodes, such as XML <textview> 、 <imgview> 、

[0124] <videoview>The interface is defined by nodes that specify the controls contained in the interface. One node corresponds to one control or property in the interface, and the nodes are parsed and rendered to present the content visible to the user. In addition, many applications, such as hybrid applications, also contain web pages in the interface. A web page, also referred to as a page, can be understood as a special control embedded in the interface of an application. The web page is a source code written in a specific computer language, such as hyper text markup language (HTML), cascading stylesheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed by a browser or a web page display component similar to the function of a browser to present content recognizable to the user. The specific content contained in the web page is also defined by tags or nodes in the web page source code, such as HTML defines the content of the web page by tags such as 、 、 <video> 、 <canvas>Used to define the elements and attributes of a webpage.

[0125] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0126] As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the listed items. As used in the above embodiments, depending on the context, the term "when" can be interpreted as meaning "if..." or "after..." or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining..." or "in response to determining..." or "when (the stated condition or event) is detected" or "in response to detecting (the stated condition or event)."

[0127] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned method embodiments can be implemented by a computer program instructing relevant hardware to complete, the program can be stored in a computer-readable storage medium, and the program can include the processes of the above-mentioned method embodiments when executed. The aforementioned storage medium includes ROM or random storage memory RAM, magnetic disc or optical disc, and various storage medium capable of storing program codes.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A focusing method applied to electronic devices, characterized in that, The method comprises: acquiring a first image collected by a camera, and determining a first region as a focus region of the first image; acquiring a second image collected by the camera, determining a first candidate focus region from N2 candidate focus regions corresponding to N2 objects in the second image when the second image comprises the N2 candidate focus regions, wherein N2≥1, and an intersection-over-union (IoU) max of the first candidate focus region and the first region is maximum among the N2 candidate focus regions; determining a second region according to the first region and the first candidate focus region, and determining the second region as a focus region of the second image.

2. The method of claim 1, wherein, The determination of the second region according to the first region and the first candidate focus region specifically comprises: when the IoUmax is greater than a first threshold M1, determining the second region according to the first region and the first candidate focus region.

3. The method of claim 2, wherein, The method further comprises: when the IoUmax is less than or equal to M1, increasing a count value C of a first counter; when the C is greater than a second threshold M2, determining the first candidate focus region as the focus region of the second image; when the C is less than or equal to M2, determining the second region according to the first region and the first candidate focus region, and determining the second region as the focus region of the second image.

4. The method of claim 2, wherein, The method further comprises: when the IoUmax is less than or equal to M1, increasing a count value C of a first counter; when the C is greater than a second threshold M2, determining the first candidate focus region as the focus region of the second image; when the C is less than or equal to M2, determining the first region as the focus region of the second image.

5. The method of claim 2, wherein, The method further comprises: when the IoUmax is less than or equal to M1, determining the first candidate focus region as the focus region.

6. The method of claim 1, wherein, The determination of the second region according to the first region and the first candidate focus region specifically comprises: determining the second region according to the first region, the first candidate focus region and the IoUmax.

7. The method of claim 6, wherein, The determination of the second region according to the first region, the first candidate focus region and the IoUmax specifically comprises: St=Sx*(1-IoUmax)+S1*IoUmax; wherein S1 represents coordinates of the first region, Sx represents coordinates of the first candidate focus region, and St represents coordinates of the second region.

8. The method of claim 6, wherein, The determination of the second region according to the first region, the first candidate focus region and the IoUmax specifically comprises: St=Sx*IoUmax+S1*(1-IoUmax); wherein S1 represents coordinates of the first region, Sx represents coordinates of the first candidate focus region, and St represents coordinates of the second region.

9. The method of claim 1, wherein, The method further comprises: displaying a shooting preview interface; displaying the first image in the shooting preview interface, and displaying a first mark on the first region of the first image; displaying the second image in the shooting preview interface, and displaying a second mark on the second region of the second image.

10. The method of claim 1, wherein, The method further comprises: determining the first region as a focus region of the second image when the second image has no candidate focus region.

11. An electronic device, comprising: comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are configured to store a computer program that, when executed by the one or more processors, causes performance of the method of any one of claims 1-10.

12. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is configured to invoke computer instructions to cause the electronic device to perform the method of any one of claims 1-10.

13. A computer program product comprising instructions, characterized in that, The computer program product, when run on an electronic device, causes the electronic device to perform the method of any one of claims 1-10.

14. A computer readable storage medium comprising a computer program, characterized in that, The computer program, when run on an electronic device, causes the electronic device to perform the method of any one of claims 1-10.

Images