Focus tracking method and electronic device

Abstract

The application provides a focusing method and an electronic device. The focusing method provided by the application is implemented by an electronic device, such as a mobile phone or a tablet computer, which is provided with a camera and has a photographing function. After obtaining a latest image, the electronic device can identify one or more specific objects in the image, determine a preferred object with the highest overlap degree with a photographing focus point of a previous image from the one or more specific objects, and then determine a target object according to the photographing focus point in the previous image and the preferred object in the latest image, and determine the target object as the photographing focus point of the latest image. In this way, the change of the photographing focus point between the previous image and the latest image is smoother, and the jumping of the image content between clear and blurred can be avoided. The user can obtain a smoother and more comfortable automatic focusing photographing experience.

Description

Technical Field

[0001] This application relates to the field of terminals, and more particularly to focusing methods and electronic devices. Background Technology

[0002] Currently, electronic devices with shooting capabilities typically also offer autofocus functionality. When autofocus is enabled, the electronic device can identify specific objects in the image captured by the camera, such as people, animals, flowers, and buildings. Then, the electronic device can adjust the camera's focal length based on the depth of field of the identified object to ensure the object is in focus. In one type of focus tracking method, the electronic device defaults to identifying the object with the highest confidence level among one or more identified objects as the focus point. However, this can lead to frequent focus jumps, negatively impacting the user's shooting experience. Summary of the Invention

[0003] This application provides a focus tracking method and an electronic device.

[0004] In a first aspect, this application provides a focus tracking method applied to an electronic device. The method includes: acquiring a first image captured by a camera and determining a first region as the focus region of the first image; acquiring a second image captured by the camera, wherein when the second image includes N2 candidate focus regions corresponding to N2 objects, a first candidate focus region is determined from the N2 candidate focus regions, where N2≥1 and the intersection-union ratio (IoUmax) between the first candidate focus region and the first region is maximized; and determining a second region based on the first region and the first candidate focus region, and determining the second region as the focus region of the second image.

[0005] By implementing the above method, after acquiring the latest frame image, the electronic device can identify one or more specific objects in the image, determine one or more candidate focus areas corresponding to these one or more specific objects, and then determine the candidate focus area (i.e., the first focus area) with the highest overlap with the focus area of ​​the previous frame image (i.e., the focus area of ​​the previous frame image) from these one or more candidate focus areas. Then, the focus area (second area) of the second image is determined based on the focus area of ​​the previous frame image and the aforementioned first focus area.

[0006] This results in a smoother change in focus between consecutive frames, avoiding the jerky transitions between sharp and blurry images caused by focus jumps. Users can enjoy a more fluid and comfortable autofocus shooting experience.

[0007] In some embodiments, determining a second region based on a first region and a first candidate focus region specifically includes: determining a second region based on a first region, a first candidate focus region, and IoUmax.

[0008] In some embodiments, the second region is determined based on the first region, the first candidate focus region, and IoUmax, specifically including: St=Sx*(1-IoUmax)+S1*IoUmax; where S1 represents the coordinates of the first region, Sx represents the coordinates of the first candidate focus region, and St represents the coordinates of the second region.

[0009] In other embodiments, the second region is determined based on the first region, the first candidate focus region, and IoUmax, specifically including: St=Sx*IoUmax+S1*(1-IoUmax); where S1 represents the coordinates of the first region, Sx represents the coordinates of the first candidate focus region, and St represents the coordinates of the second region.

[0010] The second region, determined by the above method, has a smoother transition with the first region, which helps to avoid focus jumps and thus avoids image content jumping between sharpness and blur, giving users a smoother and more comfortable autofocus shooting experience.

[0011] In some embodiments, determining a second region based on a first region and a first candidate focus region specifically includes: when IoUmax > a first threshold M1, determining a second region based on the first region and the first candidate focus region.

[0012] In some embodiments, the method further includes: when IoUmax≤M1, increasing the count value C of the first counter; when C>the second threshold M2, determining the first candidate focus region as the focus region of the second image; when C≤M2, determining the second region based on the first region and the first candidate focus region, and determining the second region as the focus region of the second image.

[0013] In some embodiments, the method further includes: when IoUmax≤M1, increasing the count value C of the first counter; when C>the second threshold M2, determining the first candidate focus region as the focus region of the second image; when C≤M2, determining the first region as the focus region of the second image.

[0014] In some embodiments, the method further includes: when IoUmax≤M1, determining the first candidate focus region as the focus region.

[0015] By implementing the above method, the electronic device can use the relationship between IoUmax and the preset threshold M1 to adopt different specific methods for determining the focus in the new image, thereby achieving different focus tracking effects while avoiding frequent changes in the focus.

[0016] In some embodiments, the method further includes: displaying a shooting preview interface; displaying a first image in the shooting preview interface, wherein a first mark is displayed on a first region of the first image; and displaying a second image in the shooting preview interface, wherein a second mark is displayed on a second region of the second image.

[0017] In this way, users can determine the current shooting focus in real time, so that they can adjust it at any time to meet their changing shooting needs.

[0018] In some embodiments, the method further includes: when the second image does not have a candidate focus area, determining the first area as the focus area of ​​the second image.

[0019] In a second aspect, this application provides an electronic device including one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store a computer program, which, when executed by one or more processors, causes the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0020] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0021] Fourthly, this application provides a computer-readable storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fifthly, this application provides a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0024] Figures 1A-1B This is a set of user interface diagrams for shooting focus jump provided in the embodiments of this application; Figure 2This is a flowchart of the focus tracking method provided in the embodiments of this application; Figure 3 This is a schematic diagram of determining the region IoU based on region coordinates provided in an embodiment of this application; Figure 4 This is a schematic diagram of the target object provided in the embodiments of this application; Figure 5 This is a flowchart of another focus tracking method provided in the embodiments of this application; Figures 6A-6B This is a set of focus tracking schematic diagrams provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0026] Currently, electronic devices with shooting capabilities typically also offer autofocus functionality. When autofocus is enabled, the electronic device can identify specific objects in the image captured by the camera, such as people, animals, flowers, buildings, etc. Then, the electronic device can adjust the camera's focal length based on the depth of field of the identified object, ensuring that the object is in focus. This specific object is the focal point. The area where the focal point is located is the focus area. In a video stream, the action of determining the focal point of each frame is called focus tracking.

[0027] Typically, a single image frame may contain multiple specific objects, such as multiple people, people and animals, or people and buildings. In this case, the electronic device needs to select one of these objects as the focus point. In one autofocus tracking method, the electronic device defaults to identifying the object with the highest confidence among one or more detected objects as the focus point. However, this can easily lead to frequent focus point shifts, negatively impacting the user's shooting experience.

[0028] Figures 1A-1B This is a set of user interface diagrams for shooting focus jump provided in the embodiments of this application.

[0029] refer to Figure 1AAfter enabling autofocus, the electronic device can use an object recognition algorithm to identify objects 11, 12, and 13 in image 10 captured by the camera. Assume the confidence levels of objects 11, 12, and 13 output by the object recognition algorithm are C1, C2, and C3, respectively, where C1 > C2 > C3. Based on the confidence levels from highest to lowest, the electronic device can determine object 11, with the highest confidence level, as the focus point. Therefore, the electronic device can adjust the camera's focal length according to the depth of field of object 11, making object 11 appear sharp. At this point, objects 12 and 13, which are not within the same depth of field as object 11, will appear blurry.

[0030] Optionally, after determining the shooting focus, the electronic device may display a marker box on the shooting focus to indicate to the user that the object in the marker box is the current shooting focus. For example... Figure 1A As shown, after determining that the object 11 is the focus of the shot, the electronic device can display a marker box 101 on the object 11 to indicate to the user that the current focus of the shot is the object 11. The area covered by the marker box 101 is also the focus area of ​​the image 10, and the marker box 101 is also used to indicate the focus area of ​​the image 10.

[0031] refer to Figure 1B Following image 10, the camera can capture image 20. Similarly, based on the object recognition algorithm, the electronic device can identify objects 21, 22, and 23 in image 20, with confidence levels of C4, C5, and C6 respectively, where C5 > C4 > C6. At this point, the electronic device will determine object 22, with the highest confidence level, as the new focus point and adjust the camera's focal length according to the depth of field of object 22, making object 22 appear sharp. Objects 21 and 23, which are not within the same depth of field as object 22, will appear blurry. Similarly, after determining object 22 as the new focus point, the electronic device can display a marker box 201 on object 22 to indicate to the user that the current focus is on object 22. The area covered by the marker box 201 is also the focus area of ​​image 20, and the marker box 201 is used to indicate the focus area of ​​image 20.

[0032] like Figures 1A-1B As shown, during the process of switching the shooting focus from object 11 in the lower left region of the image to object 22 in the upper right region of the image, the sharp image area where the shooting focus is located undergoes a significant jump.

[0033] Similarly, the focus tracking method described above, which determines the shooting focus solely based on the level of confidence, does not involve the content relationship between the preceding and following images. This can easily lead to frequent changes in the shooting focus, causing the image content at the same location to be sometimes clear and sometimes blurry, thus affecting the user's shooting experience.

[0034] Therefore, this application provides a focus tracking method. This method can be applied to electronic devices such as mobile phones and tablets equipped with cameras and capable of taking pictures. By implementing the above focus tracking method, after acquiring the latest frame image, the electronic device can identify one or more specific objects in the image, determine the preferred object from these one or more specific objects that has the highest overlap with the focus of the previous frame image, and then determine the target object based on the focus of the previous frame image and the preferred object in the latest frame image, thus setting the target object as the focus of the latest frame image.

[0035] This results in a smoother change in focus between two consecutive frames, avoiding the jerky transitions between sharp and blurry images caused by focus jumps. Users can enjoy a smoother and more comfortable autofocus shooting experience.

[0036] Not limited to mobile phones and tablets, electronic devices implementing the above-described focusing method can also be action cameras, drones, augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, and / or smart city devices, etc. This application does not impose any special limitations on the type of electronic device.

[0037] Figure 2 This is a flowchart of the focus tracking method provided in the embodiments of this application.

[0038] S211, Obtain the first frame image F1.

[0039] In some embodiments, after the camera is turned on, the electronic device automatically enables autofocus by default. The first frame captured by the camera is then designated as the first frame image F1.

[0040] In some embodiments, the electronic device disables autofocus by default after the camera is turned on. Upon detecting a user action that enables autofocus, the electronic device enables autofocus. At this point, the first frame captured by the camera after enabling autofocus is designated as the first frame image F1.

[0041] S212. Determine N1 objects in F1, where N1 ≥ 1? Electronic devices can process F1 through a preset object recognition algorithm to obtain specific objects in F1.

[0042] When F1 includes at least one (i.e., N1, N1≥1) preset specific object (e.g., people, animals, flowers, buildings, etc.), the electronic device immediately executes S213. Conversely, if no specific object is recognized from F1, the electronic device acquires the next frame image captured by the camera and executes S212 again until at least one preset specific object is recognized, and then executes S213.

[0043] S213. Determine the object with the highest confidence among N1 objects as the shooting focus, and adjust the camera focal length according to the depth of field of the shooting focus to make the shooting focus image clear.

[0044] S214, Get the next frame image F2.

[0045] S215. Determine N2 objects in F2, where N2 ≥ 1? Similarly, when F2 includes at least one (i.e., N2, N2≥1) preset specific objects, the electronic device immediately executes S216. Conversely, if no specific object is identified in F2, the electronic device can use the coordinates of the shooting focus in the previous frame image to determine the image content at the corresponding position in the current frame image F2 as the shooting focus, and then adjust the camera focal length according to the depth of field of the new shooting focus to make the shooting focus in the current frame image F2 clear, and continue to acquire the next frame image, and execute S215 again until at least one preset specific object is identified, and then execute S216.

[0046] S216. Determine the preferred object among N2 objects that has the highest overlap with the shooting focus of the previous frame image.

[0047] Electronic devices can represent the degree of overlap between two regions using the Intersection of Union (IoU). A higher IoU indicates a greater degree of overlap, while a lower IoU indicates a lower degree of overlap. 0 ≤ IoU ≤ 1.

[0048] The electronic device can obtain the coordinates of the focus point in the previous frame image and the coordinates of each object in the current frame image. Based on the coordinates, it determines the IoU (Intersection over Union) between each object in the current frame image and the focus point in the previous frame image, i.e., the degree of overlap.

[0049] Figure 3 This is a schematic diagram illustrating the determination of regional IoU based on regional coordinates, provided in an embodiment of this application. For example... Figure 3As shown, dashed box 31 represents the area where the focus was located in the previous frame, and solid box 32 represents the area where an object is located in the current frame. For any rectangle, its position can be represented by the coordinates of its diagonal vertices. For example, dashed box 31 can be represented by coordinates S1(X1,Y1,X2,Y2), and solid box 32 can be represented by coordinates S2(X3,Y3,X4,Y4). Here, (X1,Y1) and (X2,Y2) are a pair of diagonal vertices of dashed box 31, and (X3,Y3) and (X4,Y4) are a pair of diagonal vertices of solid box 32.

[0050] Therefore, the intersection I2 of the dashed box 31 and the solid box 32 can be represented as: I2 = (X2 - X3)(Y2 - Y3); The union U2 of dashed box 31 and solid box 32 can be represented as: U2=[(X4-X1)(Y4-Y1)-(X3-X1)(Y4-Y2)-(X4-X2)(Y3-Y1)]; The IoU2 of dashed box 31 and solid box 32 is represented as follows: IoU2=(X2-X3)(Y2-Y3) / [(X4-X1)(Y4-Y1)-(X3-X1)(Y4-Y2)-(X4-X2)(Y3-Y1)].

[0051] In some embodiments, after determining the preferred object among N2 objects that has the highest overlap with the shooting focus in the previous frame image, the electronic device can determine the preferred object as the shooting focus of the current frame image, and then adjust the camera focal length according to the depth of field of the preferred object to make the preferred object clear in image.

[0052] The preferred object is the one with the highest overlap with the focus point in the previous frame, i.e., the object closest to the focus point in the previous frame. Within a short timeframe, the position of the subject within the frame generally doesn't change. Therefore, the object closest to the focus point in the previous frame is most likely the same object that was the focus point in that frame. By implementing this focus tracking method, the electronic device can automatically and continuously focus on the same object, performing the shooting operation and obtaining stable and smooth video, avoiding focus point jumps and the resulting local image jumps.

[0053] Preferably, in this embodiment of the application, after determining the preferred object with the highest overlap with the shooting focus in the previous frame image among the N2 objects, that is, after executing S216, the electronic device can execute S217 to further avoid shooting focus jump.

[0054] S217. Determine the target object based on the shooting focus of the previous frame and the preferred object of the current frame, and set the target object as the shooting focus of the current frame.

[0055] In one embodiment, the electronic device can determine the coordinates St of the target object according to the following formula (1). St = Sx * (1 - IoUmax) + S1 * IoUmax; Where Sx represents the coordinates of the preferred object in the current frame image, such as S2(X3,Y3,X4,Y4) shown in S216, S1 represents the coordinates of the shooting focus in the previous frame image, such as S1(X1,Y1,X2,Y2), and IoUmax represents the intersection-over-union ratio of the preferred object and the shooting focus in the previous frame.

[0056] Assuming, Figure 3 The object shown in the solid box 32 is the preferred object with the highest overlap with the focus of the previous frame image, so IoUmax = IoU2. Substituting S1(X1,Y1,X2,Y2), S2(X3,Y3,X4,Y4) and IoU2 into formula (1) yields the coordinates St of the target object: St=(X3,Y3,X4,Y4)*(1-IoU2)+(X1,Y1,X2,Y2)*IoU2; =(X5,Y5,X6,Y6); Therefore, the electronic device can determine that the image content corresponding to St(X5,Y5,X6,Y6) in the current frame image is the target object.

[0057] Figure 4 This is a schematic diagram of the target object provided in the embodiments of this application. For example... Figure 4 As shown, the image content enclosed by the solid-line box 33 corresponding to St(X5,Y5,X6,Y6) in the current frame image is the target object. Therefore, the electronic device can adjust the camera focal length according to the depth of field of the target object to make the target object clear.

[0058] After executing S217, the electronic device can acquire the next frame image, such as the third frame image F3 captured by the camera. The operations shown in S214-S217 are performed on F3 to determine the preferred object in F3 with the highest overlap of shooting focus with the previous frame image F2, thereby determining the target object of F3 and determining the shooting focus of F3. These details will not be elaborated here.

[0059] Compared to Figure 3 The method shown directly sets the preferred object as the shooting focus is... Figure 4The method shown here, which determines the target object based on the shooting focus of the previous frame and the preferred object of the current frame, and then sets the target object as the shooting focus, can further constrain the range of shooting focus changes, avoid shooting focus jumps, and thus avoid local image jumps caused by shooting focus jumps, allowing users to obtain more stable and smoother videos.

[0060] In other embodiments, the electronic device may also determine the coordinates St of the target object according to the following formula (2). St = Sx * IoUmax + S1 * (1 - IoUmax); According to formula (2), electronic devices can also further constrain the range of shooting focus changes, avoid shooting focus jumps, and enable users to obtain more stable and smoother videos.

[0061] Figure 5 This is a flowchart of another focus tracking method provided in the embodiments of this application.

[0062] like Figure 5 As shown, after S216 and before S217, preferably, the electronic device can also determine whether the intersection-over-union ratio (IoU-max) between the preferred object and the focus of the previous frame is greater than the threshold M1. When IoU-max > M1, the electronic device immediately executes S217. IoU-max ≤ M1 means that the preferred object does not overlap with the focus of the previous frame or the overlapping area is too small, that is, the distance between the preferred object and the focus of the previous frame is not close enough.

[0063] In some embodiments, when IoU-max≤M1, the electronic device can use the coordinates of the shooting focus in the previous frame image to determine the image content at the corresponding position in the current frame image as the shooting focus, and then adjust the camera focal length according to the depth of field of the new shooting focus to make the shooting focus in the current frame image clear. Then, the electronic device continues to acquire the next frame image and executes S215 and the subsequent process again.

[0064] In some embodiments, when IoU-max≤M1, the electronic device may also directly set the preferred object as the shooting focus of the current frame.

[0065] Preferably, in other embodiments, such as Figure 5As shown, when IoU-max ≤ M1, the electronic device can increment the counter value. Then, the electronic device can determine whether the counter value coun is greater than the threshold M2. If coun > M2, it means that for a relatively long period, the preferred object in each frame is far from the previously determined shooting focus, and the previous shooting focus no longer meets the user's shooting needs. Therefore, when coun ≤ M2, the electronic device can execute S217, determining the target object based on the shooting focus of the previous frame and the preferred object in the current frame, and setting the target object as the shooting focus of the current frame. When coun > M2, the electronic device can directly set the preferred object as the shooting focus of the current frame.

[0066] Alternatively, in some embodiments, when coun ≤ M2, the electronic device may also use the coordinates of the shooting focus of the previous frame to determine the image content at the corresponding position in the current frame as the shooting focus. When coun > M2, the electronic device may directly set the preferred object as the shooting focus of the current frame.

[0067] Figures 6A-6B This is a set of focus tracking diagrams provided in the embodiments of this application.

[0068] After the camera is turned on, it can periodically generate a frame of image based on the acquired light signal. For example, at a frame rate of 30 FPS, the camera can generate a frame of image every 33.3 ms. The continuously generated images by the camera are called a video stream.

[0069] With autofocus enabled by default, after acquiring the first frame (F1) generated by the camera, the electronic device can identify objects within that frame. (Reference) Figure 6A For example, the electronic device can identify objects T1, T2, and T3 included in F1, and simultaneously determine the confidence levels of these objects T1, T2, and T3. For example, the confidence level of T1 is 0.96, the confidence level of T2 is 0.90, and the confidence level of T3 is 0.87. In this case, the electronic device can determine that the object with the highest confidence level in F1 is T1. Therefore, the electronic device can determine T1 as the focus point and then adjust the camera's focal length according to the depth of field of T1, making the focus point T1 appear sharp. At this time, objects T2 and T3, which are not within the same depth of field as the focus point T1, appear blurry.

[0070] Subsequently, the electronic device acquires the second frame image F2 generated by the camera. Similarly, the electronic device first identifies the objects in F2. For example, the electronic device can identify objects T5, T6, and T7 in F2, and simultaneously determine the confidence level of each object. For example, the confidence level of T5 is 0.95, the confidence level of T6 is 0.96, and the confidence level of T7 is 0.87. At this point, if the conventional method of determining the shooting focus based on confidence level is used, the electronic device would determine the object T6 with the highest confidence level as the shooting focus.

[0071] However, implementation Figure 2 The method shown allows the electronic device to determine the overlap between objects T5, T6, and T7 and the focus point in the previous frame image after identifying objects T5, T6, and T7; that is, the Intersection over Union (IoU) between objects T5, T6, and T7 and the focus point in the previous frame image. The IoU ratio of T5 to the focus point in the previous frame image is denoted as IoU5, T6 as IoU6, and T7 as IoU7. For example... Figure 6A As shown, the dashed box 601 in F2 represents the focus point T1 in the previous frame. Obviously, the electronic device can determine that IOU5 > IOU6 > IOU7, and thus determine that T5 is the object with the highest overlap with the focus point in the previous frame.

[0072] Therefore, the electronic device can determine T5 as the preferred object. Then, the electronic device can determine the target object (the image content shown in the dashed box 602) based on the shooting focus T1 in the previous frame image F1 and the preferred object T5, set the target object as the shooting focus of F2, and then adjust the camera focal length according to the depth of field of the shooting focus to make the image of the shooting focus clear. The above process can be referred to in detail in the description of S217, and will not be repeated here.

[0073] refer to Figure 6B After acquiring the third frame image F3 generated by the camera, the electronic device first identifies the objects in F3. For example, the electronic device may identify objects T8 and T9 in F3 and simultaneously determine the confidence levels of these objects, such as 0.94 and 0.93. Then, the electronic device may identify the preferred object from among objects T8 and T9 that has the highest overlap with the focus of the previous frame image F2, for example, T8. Then, the electronic device may determine the target object (the image content shown in dashed box 603) based on the focus of the previous frame image F2 (i.e., the image content shown in dashed box 602) and the preferred object T8, set the target object as the focus of F3, and then adjust the camera focal length according to the depth of field of the focus to ensure a clear image of the focus.

[0074] Similarly, electronic devices can constrain the range of changes in the shooting focus, avoiding focus jumps and thus preventing local image jumps caused by focus jumps, resulting in more stable and smoother videos and improving the user experience.

[0075] exist Figures 6A-6B In the example shown, the second frame image F2 can be referred to as the first image, and the area enclosed by the dashed box 602 can be referred to as the first region; the third frame image F3 can be referred to as the second image, and the areas where objects T8 and T9 are located in F3 are two candidate focus areas. Among them, the area where T8 has the highest overlap with the shooting focus (i.e., the first region) in the previous frame image F2 is the first candidate focus area, and the area enclosed by the dashed box 603 is the second region.

[0076] refer to Figures 1A-1B For the shooting focus, the electronic device can display a marker box on the shooting focus. Therefore, in the shooting preview interface, when F2 is displayed, the electronic device can display a marker box (first marker) at the position of the dashed frame 602, indicating that the image content in the dashed frame 602 is the shooting focus, that is, indicating that the image content in the dashed frame 602 is the focus area; similarly, when F3 is displayed, the electronic device can display a marker box (second marker) at the position of the dashed frame 603, indicating that the image content in the dashed frame 603 is the shooting focus.

[0077] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0078] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0079] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0080] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0081] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0082] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0083] For example, processor 110 can couple to touch sensor 180K via I2C interface, enabling communication between processor 110 and touch sensor 180K via I2C bus interface to realize the touch function of electronic device; processor 110 can couple to audio module 170 via I2S bus to realize communication between processor 110 and audio module 170; audio module 170 can transmit audio signals to wireless communication module 160 via I2S interface to realize the function of answering phone calls through Bluetooth headset; audio module 170 can also transmit audio signals to wireless communication module 160 via PCM interface to realize the function of answering phone calls through Bluetooth headset; audio module 170 can also transmit audio signals to wireless communication module 160 via UART interface to realize the function of playing music through Bluetooth headset. MIPI interface includes camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, processor 110 and camera 193 communicate via CSI interface to realize the shooting function of electronic device. Processor 110 and display 194 communicate via DSI interface to realize the display function of electronic device. USB port 130 can be used to connect a charger to charge electronic devices, or to transfer data between electronic devices and peripheral devices. It can also be used to connect headphones to play audio.

[0084] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0085] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also power electronic devices via the power management module 141. The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc.

[0086] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0087] Mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G for use in electronic devices. Wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies for use in electronic devices. In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the electronic device to communicate with networks and other devices via wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0088] Electronic devices utilize GPUs, displays (194), and application processors to achieve display functions. Figures 1A-1B The shooting preview interface shown.

[0089] The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU performs mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information. The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD). The display panel may also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microled, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0090] Electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0091] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP processes the data fed back by camera 193 and converts it into an image visible to the naked eye. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for further processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0092] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0093] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0094] RAM can include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), and double data rate synchronous dynamic random-access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM). NVM can include disk storage devices and flash memory.

[0095] RAM can be directly read and written by processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data. NVM can also store executable programs and user and application data, and can be pre-loaded into RAM for direct reading and writing by processor 110.

[0096] In this embodiment, the executable program code implementing the focus tracking method described in this embodiment can be stored in the NVM. After the electronic device is powered on, the electronic device can load the executable program code stored in the NVM into the RAM, so that the electronic device can automatically and continuously focus on the same object, perform shooting operations, obtain stable and smooth video, and avoid focus jumps and local image jumps caused by focus jumps.

[0097] The external memory interface 120 can be used to connect to an external NVM to expand the storage capacity of the electronic device. In some embodiments, the executable program code implementing the focus tracking method described in this application can also be stored in the external NVM connected to the external memory interface 120.

[0098] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0099] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Electronic devices can receive button input via buttons 190, generating key signal inputs related to user settings and function control. Motor 191 can generate vibration alerts. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. SIM card interface 195 is used to connect a SIM card.

[0100] The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0101] A pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A may be located on the display screen 194. When a touch operation is applied to the display screen 194, the electronic device detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device can also calculate the touch position based on the detection signal from the pressure sensor 180A. A gyroscope sensor 180B can be used to determine the angular velocity of the electronic device around three axes (i.e., x, y, and z axes). An accelerometer sensor 180E can be used to detect the magnitude of the acceleration of the electronic device in various directions (generally three axes). The electronic device can determine its motion posture using the gyroscope sensor 180B and the accelerometer sensor 180E. In shooting scenarios, the gyroscope sensor 180B can also be used for image stabilization. A barometric pressure sensor 180C is used to measure barometric pressure. A magnetic sensor 180D includes a Hall effect sensor. The electronic device can use the magnetic sensor 180D to detect the opening and closing of the flip cover, and then set features such as automatic flip unlocking based on the detected opening and closing state of the cover or the flip cover. The distance sensor 180F is used to measure distance. In a shooting scenario, the electronic device can use the distance sensor 180F to determine the depth of field of the subject, and then adjust the focal length of the camera 193 based on this depth of field to achieve fast focusing. The proximity sensor 180G can be used for object detection. The ambient light sensor 180L is used to sense ambient light brightness. The electronic device can adaptively adjust the brightness of the display 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect temperature. The bone conduction sensor 180M can acquire vibration signals.

[0102] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can then transmit the detected touch operation to the application processor to determine the type of touch event. Subsequently, the electronic device can provide visual output related to the touch operation through display screen 194.

[0103] The term "user interface (UI)" used in the specification, claims, and drawings of this application refers to the medium through which an application or operating system interacts and exchanges information with the user. It converts the internal form of information into a form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content, such as images, text, buttons, and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined through tags or nodes, such as in XML. <textview> 、 <imgview> 、 <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a web page is also defined through tags or nodes in the web page source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>Used to define the elements and attributes of a webpage.

[0104] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an interface element such as an icon, window, or control displayed on the screen of an electronic device. The control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0105] As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the listed items. As used in the above embodiments, depending on the context, the term "when" can be interpreted as meaning "if..." or "after..." or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining..." or "in response to determining..." or "when (the stated condition or event) is detected" or "in response to detecting (the stated condition or event)."

[0106] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0107] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A focusing method applied to electronic devices, characterized in that, The method includes: Acquire the first image captured by the camera, and determine the first region as the focus area of ​​the first image; Acquire a second image captured by the camera. When the second image includes N2 candidate focus regions corresponding to N2 objects, determine a first candidate focus region from the N2 candidate focus regions. N2 ≥ 1, and the intersection-union ratio (IoUmax) between the first candidate focus region and the first region is maximized among the N2 candidate focus regions. The second region is determined based on the first region and the first candidate focus region; Determining the second region based on the first region and the first candidate focus region includes: The second region is determined based on St=Sx*(1-IoUmax)+S1*IoUmax or St=Sx*IoUmax+S1*(1-IoUmax), and the second region is determined as the focus region of the second image; where S1 represents the coordinates of the first region, Sx represents the coordinates of the first candidate focus region, and St represents the coordinates of the second region.

2. The method according to claim 1, characterized in that, The step of determining the second region based on the first region and the first candidate focus region specifically includes: When IoUmax > the first threshold M1, the second region is determined based on the first region and the first candidate focus region.

3. The method according to claim 2, characterized in that, The method further includes: When IoUmax≤M1, increase the count value C of the first counter; When C > the second threshold M2, the first candidate focus region is determined to be the focus region of the second image; When C≤M2, a second region is determined based on the first region and the first candidate focus region, and the second region is determined as the focus region of the second image.

4. The method according to claim 2, characterized in that, The method further includes: When IoUmax≤M1, increase the count value C of the first counter; When C > the second threshold M2, the first candidate focus region is determined to be the focus region of the second image; When C≤M2, the first region is determined to be the focus region of the second image.

5. The method according to claim 2, characterized in that, The method further includes: when IoUmax≤M1, determining the first candidate focus region as the focus region.

6. The method according to claim 1, characterized in that, The method further includes: Displays the shooting preview interface; The first image is displayed in the shooting preview interface, and a first mark is displayed on the first area of ​​the first image; The second image is displayed in the shooting preview interface, and a second mark is displayed on the second area of ​​the second image.

7. The method according to claim 1, characterized in that, The method further includes: When the second image does not have a candidate focus area, the first area is determined as the focus area of ​​the second image.

8. An electronic device, characterized in that, It includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program that, when the one or more processors execute the computer program, causes the method as described in any one of claims 1-7 to be performed.

9. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-7.

10. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-7.

11. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-7.

Images