An image processing method, apparatus, electronic device, and storage medium

Abstract

This disclosure relates to an image processing method applied to an electronic device, the electronic device having multiple image acquisition modules, comprising: determining N focus targets within a framing area; determining M image acquisition modules from the plurality of image acquisition modules based on a first distance between the N focus targets and each of the image acquisition modules, at least one of the image features of the N focus targets, and structural attribute parameters of the image acquisition modules; M being less than or equal to N; controlling the M image acquisition modules to focus on and acquire images of the N focus targets respectively, wherein each of the M image acquisition modules corresponds to a different focus target; and fusing the first images acquired by each of the M image acquisition modules to generate a second image. This method improves the focusing effect and image acquisition effect for different focus targets, thereby improving the quality of the final fused image.

Description

Technical Field

[0001] This disclosure relates to the field of image technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] With the widespread use and application of electronic devices, electronic products are supporting more and more applications and becoming more and more powerful. Furthermore, electronic devices are developing towards diversification and personalization, providing more personalized and professional services, and have become indispensable products in life, such as cameras, mobile phones, and tablets.

[0003] Electronic devices such as cameras, mobile phones, and tablets all have a photography function. With the diversification of shooting scenarios, the shooting targets also include multiple different targets, which may vary in size, distance, and other factors. Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0005] A first aspect of this disclosure provides an image processing method applied to an electronic device, the electronic device having multiple image acquisition modules, comprising: determining N focus targets within a framing area; determining M image acquisition modules among the multiple image acquisition modules based on a first distance between the N focus targets and each of the image acquisition modules, at least one of the image features of the N focus targets, and structural attribute parameters of the image acquisition modules; M being less than or equal to N; controlling the M image acquisition modules to focus on and acquire images of the N focus targets respectively, wherein each of the M image acquisition modules corresponds to a different focus target; and fusing the first images acquired by each of the M image acquisition modules to generate a second image.

[0006] In one embodiment, determining M image acquisition modules from the plurality of image acquisition modules based on the distances between the N focus targets and each of the image acquisition modules, at least one of the image features of the N focus targets, and structural attribute parameters of the image acquisition modules includes: determining K image acquisition modules from the plurality of image acquisition modules based on the distances between each focus target and each of the image acquisition modules, and the depth of field of each of the image acquisition modules; wherein, the distances between each focus target and the K image acquisition modules are at least equal to the distances between the N focus targets and each of the K image acquisition modules. Within the depth of field of one of the image acquisition modules; K is greater than or equal to M; K, M, and N are all positive integers; sort the N focus targets by area according to the area corresponding to the edge contour of each focus target, and determine the first sorting result of the N focus targets; sort the K image acquisition modules by field of view according to the field of view of the K image acquisition modules, and determine the second sorting result of the K image acquisition modules; based on the first sorting result and the second sorting result, determine the M image acquisition modules from the K image acquisition modules that focus on and acquire images of the N focus targets respectively.

[0007] In one embodiment, the step of sorting the N focus targets by area based on the area corresponding to the edge contours of each focus target to determine a first sorting result includes: determining the area sorting of the edge contours of each focus target; determining that at least two edge contours have the same area size; sorting the focus targets by distance based on the distance between the at least two focus targets with the same area size and the center position of the image where the N focus targets are located, to obtain a third sorting result; and determining the first sorting result based on the area sorting and the third sorting result.

[0008] In one embodiment, determining the M image acquisition modules from the K image acquisition modules, which are used to focus on and acquire images of the N focus targets respectively, based on the first sorting result and the second sorting result, includes: determining L image acquisition modules from the K image acquisition modules that match the P-th focus target, based on the first sorting result; wherein the distance between the P-th focus target and the L image acquisition modules is within the depth of field of the L image acquisition modules; the P-th focus target is preferentially matched with the L image acquisition modules over the (P+1)-th focus target; L is less than or equal to K; determining the image acquisition module from the L image acquisition modules that performs focusing and image acquisition on the P-th focus target, based on the second sorting result; wherein P and L are both positive integers, and P is any value from 1 to N; among the L image acquisition modules, the image acquisition module ranked higher is preferentially used to focus on and acquire images of the P-th focus target over the image acquisition module ranked lower.

[0009] In one embodiment, determining the image acquisition module for focusing and acquiring images of the P-th focusing target from the L image acquisition modules according to the second sorting result includes: determining R image acquisition modules from the L image acquisition modules, wherein the R image acquisition modules are: image acquisition modules from the L image acquisition modules excluding the image acquisition module already determined for focusing and acquiring images of the focusing target; R is a positive integer greater than or equal to zero and less than or equal to L; and determining the image acquisition module for focusing and acquiring images of the P-th focusing target from the R image acquisition modules according to the second sorting result.

[0010] In one embodiment, determining N focus targets within the framing area includes: acquiring a preview image of the target image acquisition module; determining multiple focus candidate objects based on the preview image; acquiring depth information from the preview image; and determining the N focus targets from the multiple focus candidate objects based on the depth information.

[0011] In one embodiment, determining the N focus targets from the plurality of focus candidate objects based on the depth information includes: determining a second distance between each focus candidate object and the target image acquisition module based on the depth information; and identifying the focus candidate objects among the plurality of focus candidate objects whose second distance is within a first preset distance as the same focus target.

[0012] In one embodiment, determining the focus candidates that are within the second distance and the first preset distance as the same focus target includes: determining a third distance between each of the focus candidates; and determining the focus candidates among the plurality of focus candidates whose third distance is within the second preset distance and whose second distance is within the first preset distance as the same focus target.

[0013] In one embodiment, determining a plurality of focus candidates based on the preview image includes: identifying a foreground region and a background region in the preview image; and determining the plurality of focus candidates based on the foreground region.

[0014] A second aspect of this disclosure provides an image processing apparatus applied to an electronic device, the electronic device comprising a plurality of image acquisition modules, including: a focus target determination module for determining N focus targets within a framing area; an image acquisition module determination module for determining M image acquisition modules among the plurality of image acquisition modules based on a first distance between the N focus targets and each of the image acquisition modules, at least one of the image features of the N focus targets, and structural attribute parameters of the image acquisition modules; wherein M is less than or equal to N; a control module for controlling the M image acquisition modules to focus on and acquire images of the N focus targets respectively, wherein each of the M image acquisition modules corresponds to a different focus target; and a fusion module for fusion of the first images acquired by each of the M image acquisition modules to generate a second image.

[0015] A third aspect of this disclosure provides an electronic device, comprising:

[0016] A processor and a memory for storing executable instructions capable of running on the processor, wherein:

[0017] When the processor is used to run the executable instructions, the executable instructions perform the method described in any of the preceding descriptions.

[0018] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method described in any of the preceding embodiments.

[0019] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0020] This solution identifies N focus targets within the framing area. Then, based on the first distance between each of the N focus targets and its corresponding image acquisition module, at least one of the image features of the N focus targets, and the structural attribute parameters of the image acquisition modules, M image acquisition modules are selected from a pool of modules. These M modules are then controlled to focus on and acquire images of the N focus targets respectively. Different image acquisition modules correspond to different focus targets. This process determines the most suitable image acquisition module for each focus target, enabling it to focus on that target. Finally, the first images acquired by each module are fused to generate a second image. Since each of the N focus targets has a suitable image acquisition module for focusing, the sharpness of the focus targets in the synthesized image is ensured.

[0021] In summary, the method provided in this disclosure acquires images of the focus target using an image acquisition module that best matches the focus target, resulting in better preview images for each focus target. By fusing preview images of multiple focus targets acquired by multiple image acquisition modules, the final target preview image displays images of different focus targets acquired by multiple image acquisition modules, thereby enhancing the display effect of the target preview image and improving the image display quality.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0024] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0025] Figure 2 This is a schematic diagram illustrating a process for determining N focus targets within a framing area according to an exemplary embodiment;

[0026] Figure 3 This is a schematic diagram illustrating the determination of a plurality of candidate objects according to an exemplary embodiment;

[0027] Figure 4 This is a schematic diagram of a preview image according to an exemplary embodiment;

[0028] Figure 5 This is a schematic diagram of a prominent view of a preview image according to an exemplary embodiment;

[0029] Figure 6 This is a schematic diagram illustrating a process for determining a focus target according to an exemplary embodiment;

[0030] Figure 7 This is a schematic diagram illustrating a process for determining M image acquisition modules among a plurality of image acquisition modules according to an exemplary embodiment;

[0031] Figure 8 This is a schematic diagram illustrating another process for determining a first sorting result according to an exemplary embodiment;

[0032] Figure 9 This is a flowchart illustrating, according to an exemplary embodiment, the determination of M image acquisition modules for focusing and image acquisition on N focus targets respectively;

[0033] Figure 10This is a schematic flowchart illustrating another image processing method according to an exemplary embodiment;

[0034] Figure 11 This is a schematic diagram of the structure of an image processing apparatus according to an exemplary embodiment;

[0035] Figure 12 This is a block diagram illustrating a terminal device according to an exemplary embodiment. Detailed Implementation

[0036] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0037] Typically, a client application has at least one image acquisition module, meaning it has at least one camera. With one image acquisition module (one camera), the client uses this single module to capture images of the target object. Focusing and image acquisition of all targets are performed through this single module. If there are multiple targets to focus on simultaneously, each target needs to be focused on separately, and then images are captured from each target individually.

[0038] An image acquisition module can typically only focus on and acquire images of one target at a time. The focusing and image acquisition of other targets will be poor, resulting in an image where only one target is clearly focused and displayed, while others are poorly focused and displayed. Ultimately, the overall image quality will be significantly reduced, leading to a poor overall image display.

[0039] When there are multiple focus targets and multiple image acquisition modules in the client terminal, each image acquisition module includes a camera. That is, when the client has multiple cameras, multiple image acquisition modules focus on and acquire images of different focus targets respectively. This method involves randomly assigning image acquisition modules to focus on and acquire images of different focus targets; in other words, the image acquisition module randomly focuses on and acquires images of any one of the multiple targets.

[0040] The focusing and image acquisition results obtained through this method will not be very good. The image acquisition module and the focus target for focusing and image acquisition may not be matched, which may result in poor focusing and poor image acquisition quality.

[0041] refer to Figure 1 This is a flowchart illustrating an image processing method provided by this technical solution. The data processing method includes the following steps:

[0042] Step S100: Determine N focus targets within the framing area.

[0043] Step S200: Based on the first distance between the N focus targets and each image acquisition module, at least one of the image features of the N focus targets, and the structural attribute parameters of the image acquisition modules, determine M image acquisition modules among the multiple image acquisition modules; M is less than or equal to N.

[0044] Step S300: Control M image acquisition modules to focus and acquire images of N focus targets respectively. Among the M image acquisition modules, each image acquisition module corresponds to a different focus target.

[0045] Step S400: Fuse the first images acquired by each of the M image acquisition modules to generate a second image.

[0046] This method can be executed in at least a mobile terminal with multiple image acquisition modules; that is, the subject executing this method can include at least a mobile terminal. Mobile terminals can include mobile phones, tablets, in-vehicle central control devices, wearable devices, smart devices, etc., and smart devices can further include smart office equipment and smart home devices, etc.

[0047] The mobile terminal may include multiple image acquisition modules. These multiple image acquisition modules may have different structural attribute parameters. These structural attribute parameters can be determined by the hardware and software of the image acquisition modules themselves. For example, different image acquisition modules may have different structural parameters, including: different maximum viewing angles supported by different image acquisition modules, and different supported focusing distances. For instance, the maximum viewing angle of a wide-angle lens is greater than that of a regular lens, and also greater than that of a telephoto lens. The focusing range of a telephoto lens is larger than that of a wide-angle lens.

[0048] In this embodiment, the camera included in the image acquisition module may further include a wide-angle camera, a telephoto camera, and an ultra-wide-angle camera. Each image acquisition module may include at least one camera, and the terminal device may include multiple image acquisition modules. In summary, in this embodiment, the terminal device may include multiple image acquisition modules, each image acquisition module includes at least one camera, and the structural attribute parameters of the cameras included in different image acquisition modules are different.

[0049] For example, image acquisition module 1 is the main camera, image acquisition module 2 is the wide-angle camera, and image acquisition module 3 is the scene camera. The main camera, wide-angle camera, telephoto camera, and ultra-wide-angle camera have different structural attribute parameters, which can be used to distinguish different cameras. Structural attribute parameters can include the field of view (FOV) and / or depth of field of the image acquisition module.

[0050] Of course, the terminal device can also include a Time of Flight (TOF) structure, also known as a depth camera. This TOF structure can detect the distance between objects in the image captured by the image acquisition module and the image acquisition module, thereby determining the depth information of objects within the field of view of the image acquisition module.

[0051] In one embodiment, the terminal device includes multiple image acquisition modules, each of which may include multiple cameras. The multiple cameras in each image acquisition module focus on the target and acquire images using the method in this embodiment, and then the images acquired by the multiple image acquisition modules are fused together again.

[0052] For step S100, the focus target within the framing area is determined. Here, the framing area can be the region corresponding to the image captured by the image acquisition module. The specific method for determining the focus target is not limited; any method capable of determining the focus target is within the protection scope of this embodiment.

[0053] For example, the focus target can be determined based on depth information, the distance between the target within the viewfinder and the image acquisition module, and other methods. In this embodiment, the number of focus targets is determined to be N.

[0054] For example, first identify potential focus targets, and then select the target focus object from among them. The specific process for this selection can be found in subsequent embodiments.

[0055] For step S200, the first distance between each of the N focusing targets and each image acquisition module can be determined by depth information. The depth information can be obtained by a TOF structure, or by simultaneously acquiring images of the region according to different image acquisition modules and determining the depth information by triangulation.

[0056] Image features can include at least one of the following: color features, texture features, shape features, edge contours, and spatial relationship features; of course, other image features are also possible. Image features can be determined using appropriate algorithms. For example, color features can be identified using color recognition algorithms, texture features using texture recognition algorithms, and edge contours using edge detection algorithms.

[0057] This step determines M image acquisition modules based on the first distance between each focus target and each image acquisition module, at least one of the image features of the N focus targets, and the structural attribute parameters of each image acquisition module. Each of the M image acquisition modules focuses on and acquires images of different focus targets among the N focus targets. The number of image acquisition modules included in the terminal device is greater than or equal to M.

[0058] This allows for the determination of the most suitable image acquisition module for the target focus based on the image feature information, structural attribute parameters of the image acquisition module, and the distance relationship between the target focus and the image acquisition module. Different image acquisition modules focus on different targets. This results in better display effects and higher image quality for the corresponding target focus image acquired by each image acquisition module. In other words, it improves the focusing and image acquisition effects for multiple different targets, thereby enhancing the quality of the target focus images acquired by each image acquisition module. Ultimately, the quality of the fused image is also improved.

[0059] This embodiment provides an idea for determining the focus target that best matches the image acquisition module for different image acquisition module pairs, and for acquiring images of the target. This embodiment does not limit the specific process; please refer to subsequent embodiments for the specific process.

[0060] For step S300, after determining the M image acquisition modules, control each of the M image acquisition modules to focus on the N focus targets respectively. Among the M image acquisition modules, different image acquisition modules focus on and acquire different focus targets.

[0061] In this embodiment, the number of M and N can be the same. M image acquisition modules correspond one-to-one with N focus targets, that is, one image acquisition module focuses and acquires images for one focus target.

[0062] In one embodiment, M can be less than N, and one image acquisition module can focus on and acquire images of multiple focus targets.

[0063] Among the M image acquisition modules, after different image acquisition modules focus on and acquire images of different focus targets, they can all obtain the first image corresponding to the image acquisition module.

[0064] In step S400, after M different image acquisition modules focus on and acquire images of different focus targets, corresponding first images are obtained. Each image acquisition module obtains one first image, and the first images obtained by different image acquisition modules can be different. Then, the first images obtained by each image acquisition module are fused to generate a second image. This second image can be the final image of the scheme.

[0065] The fusion method can be any kind of fusion method, such as fusing the focus target in different first images into the same image to obtain a second image.

[0066] This method utilizes different image acquisition modules to focus on different targets, generating distinct first images for each. When multiple image acquisition modules acquire images of multiple different targets, each module can select the most suitable target image for its specific focus based on the distance between the target and the module, the target's image features, and the module's structural parameters. This allows the module to acquire the image of the target most closely matched to it, resulting in better first images for each target. By fusing these first images, the final second image displays the different targets captured by each module, enhancing its display effect and improving its quality.

[0067] In one embodiment, reference Figure 2 This is a flowchart illustrating a process for determining N focus targets within a framing area. Step S100, determining the N focus targets within the framing area, may include:

[0068] Step S101: Obtain a preview image of the target image acquisition module.

[0069] Step S102: Based on the preview image, determine multiple focus candidate objects.

[0070] Step S103: Obtain the depth information of the preview image.

[0071] Step S104: Based on the depth information, determine N focus targets from multiple focus candidate objects.

[0072] In step S101, each of the multiple image acquisition modules can acquire images of the current image acquisition scene, i.e. the framing area, and obtain a preview image of the framing area corresponding to each image acquisition module. The current framing area may include multiple different focus targets.

[0073] In this embodiment, the framing area corresponding to different image acquisition modules may be different. One of the multiple image acquisition modules can be used as the target image acquisition module, such as a wide-angle camera or a telephoto camera, depending on the configuration of the terminal device.

[0074] The image corresponding to the viewfinder area captured by the target image acquisition module is used as a preview image of the target image acquisition module. This preview image is used to display and preview the image of the viewfinder area captured by the target image acquisition module. This preview image is a preview of the viewfinder area captured by the target image acquisition module.

[0075] Typically, the viewfinder area may include at least one image acquisition object. To obtain a clear image of this object, it is necessary to focus on it before acquisition. This will result in a high-quality image of the object. The image acquisition object can be a focus candidate, meaning the preview image includes at least one focus candidate; in this embodiment, multiple focus candidates are included.

[0076] The preview image can be obtained through the target image acquisition module. The target image acquisition module captures the view area, which includes multiple image acquisition objects, thus obtaining the preview image of the view area captured by the target image acquisition module.

[0077] In step S102, after obtaining the preview image, potential focus objects can be determined from it according to a preset method. The specific determination method can be determined based on the preset method. For example, potential focus objects can be determined through the saliency map of the preview image. By using a machine learning model to divide the preview image into regions, salient objects in the preview image can be identified. This machine learning model can be trained using a saliency map algorithm, and different network models can be trained according to different needs, thereby determining different potential focus objects based on the network model. The machine learning model can include neural network models, deep learning models, and non-machine learning models, etc.

[0078] Of course, other algorithms can be used to train the model, or other methods can be used to determine the focus candidate. No specific limitations are made here. As long as the focus candidate can be determined from the preview image, it is within the protection scope of this embodiment.

[0079] In addition, the saliency map is a black and white image, and the saliency map of the preview image can be obtained in many ways, such as by contour detection, binarizing the image pixels, etc.

[0080] Step S103: When acquiring the preview image, the depth information of the preview image can also be acquired. This depth information may include: the depth information of each object in the viewfinder area displayed in the preview image, where each object may be a different object displayed in the preview image. The depth information of each object in the preview image may be the distance between each object and the target image acquisition module.

[0081] Obtaining depth information from the preview image can include:

[0082] The depth information of the target image acquisition module's preview image is determined by triangulation, either by acquiring the preview image using a Time-of-Flight (TOF) structure or by using the preview image and another preview image obtained from a different image acquisition module. In other words, the depth information of the target image acquisition module's preview image is determined using the triangulation principle based on preview images acquired by two different image acquisition modules. Of course, the purpose of this step is to determine the depth information of the target image acquisition module's preview image; any other method that can determine the depth information of the target image acquisition module's preview image is within the scope of protection of this embodiment.

[0083] For step S104, after obtaining the depth information of the preview image, the focus target can be determined from multiple focus candidates based on the depth information. In this embodiment, the focus candidates determined from the preview image are filtered using the depth information to determine the focus target. In this embodiment, N focus targets are determined.

[0084] This step further narrows down the range of focus targets, identifying a more accurate focus target from the candidate focus objects. This reduces the number of candidate focus objects, thereby reducing the difficulty for the image acquisition module during focusing, and ultimately improving the accuracy of focusing and the quality of focusing and image acquisition.

[0085] The specific process of determining the focus target from the focus candidate objects can be determined based on depth information and actual needs, such as the distance between focus candidate objects, etc., which will not be limited here.

[0086] Step S100 can determine the focus targets that the M image acquisition modules need to focus on, which facilitates the subsequent execution of the focusing steps for the N focus targets.

[0087] In another embodiment, the method may further include:

[0088] After identifying N focus targets and before identifying M image acquisition modules, the distances between each of the N focus targets and each image acquisition module can be determined based on the depth information of the acquired preview images. Since the N focus targets are determined from multiple candidate focus objects, and these candidate focus objects are determined from the preview images, the depth information of each candidate focus object—that is, the distance between each candidate focus object and each image acquisition module—can be determined based on the depth information of the preview images.

[0089] There is a certain correlation between the depth information of the preview image and the distance between the candidate focus object and each image acquisition module. In this embodiment, the depth information of the preview image is the distance between the candidate focus object and each image acquisition module.

[0090] Once the depth information of each potential focus target is determined, the depth information of the N focus targets can be determined, which is the distance between the N focus targets and each image acquisition module. Since the N focus targets are determined from multiple potential focus targets, determining the depth information of the multiple potential focus targets allows us to determine the depth information of the N focus targets, which is the distance between the N focus targets and each image acquisition module.

[0091] In another embodiment, reference Figure 3 This is a flowchart illustrating a process for determining multiple candidate objects. Step S102, based on the preview image, determines multiple focus candidate objects, which may include:

[0092] Step S1021: Identify the foreground and background areas in the preview image.

[0093] Step S1022: Based on the foreground area, determine multiple focus candidates.

[0094] In this embodiment, after acquiring the preview image, foreground and background regions are identified from the preview image. The foreground region includes the object to be focused. The specific process of identifying the foreground and background regions can be achieved using an image region segmentation algorithm. Different identification methods may determine different foreground and background regions. This embodiment uses an image saliency map algorithm to obtain a saliency map of the preview image. Optionally, the saliency map may include white and black regions, where white regions are white pixel regions and black regions are black pixel regions; for example, the white region can be the foreground region and the black region can be the background region.

[0095] refer to Figure 4 This is a schematic diagram of a preview image, showing an example of two towers and a loft located between them. The saliency map of this preview image, obtained using a saliency map algorithm, is shown below. Figure 5As shown, the two towers and the attic are white areas in the salient image, while the other areas are black areas. Therefore, the white area including the two towers and the attic is the foreground area, and the remaining black areas are the background area.

[0096] In this embodiment, after identifying the foreground and background areas of the preview image, the foreground area including the object to be focused is determined as a focus candidate. This step is a preliminary process of determining the focus target from the preview image, so that the focus candidate can be further filtered to determine the focus target.

[0097] In one embodiment, a first region and a second region in the preview image can be identified. The first region includes objects to be focused that need to be distinguished according to the recognition algorithm, i.e., the region does not include the candidate focus objects. The second region is the region that does not need to be focused, i.e., the region does not include the candidate focus objects.

[0098] In another embodiment, before determining N focus targets from a plurality of focus candidates based on depth information, the method may further include:

[0099] Filter and / or merge focus candidates whose area is smaller than the preset area.

[0100] After identifying the candidate focus object, the edge contour of the candidate focus object can be determined. Based on the edge contour, the area of ​​the candidate focus object can be determined. If the area of ​​the candidate focus object is smaller than the preset area, the candidate focus object is filtered out, that is, the focus target is deleted, and the candidate focus object will not be used as the focus target in the future.

[0101] Alternatively, when the area of ​​a potential focus object is smaller than a preset area, the potential focus object with an area smaller than the preset target is merged with an adjacent potential focus object with an area larger than the preset area, based on the adjacent distance between potential focus objects. Here, "adjacent potential focus objects with an area larger than the preset area" refers to potential focus objects whose adjacent distance to the potential focus object with an area smaller than the preset area is within a preset adjacent distance. The preset area and preset adjacent distance can be set according to actual needs.

[0102] In one embodiment, a closing operation can be performed on the foreground region to determine the foreground region as a focus candidate. By performing a closing operation on the foreground region, foreground regions that meet the closing operation processing conditions can be further filtered and / or connected to obtain more accurate focus candidates for subsequent focusing.

[0103] In one embodiment, the foreground area can also be used as the focus target.

[0104] In another embodiment, reference Figure 6 This is a flowchart illustrating a process for determining a focus target. In step S104, based on depth information, N focus targets are determined from multiple candidate focus objects, which may include:

[0105] Step S1041: Determine the second distance between each candidate focus object and the target image acquisition module based on the depth information.

[0106] Step S1042: Among multiple focus candidates, the focus candidates whose second distance is within the first preset distance are determined as the same focus target.

[0107] In this embodiment, after determining the candidate focus objects, a second distance between each candidate focus object and the target image acquisition module can be determined based on the depth information of the acquired preview image. Since different candidate focus objects have different distances from the target image acquisition module, the distance between each candidate focus object and the target image acquisition module can be determined based on the depth information of the preview image; this distance is the second distance. Among the multiple candidate focus objects within a first preset distance, if the second distance corresponding to one candidate focus object is within the depth of field of the target image acquisition module, then the second distances corresponding to the other candidate focus objects will also be within the depth of field of the target image acquisition module. Here, candidate focus objects with second distances within the first preset distance are defined as the same focus target. The first preset distance can be set according to actual needs.

[0108] refer to Figure 4 and Figure 5 If both towers are within the first preset distance from the target image acquisition module at a second distance, then both towers are identified as the same focusing target. If the attic is not within the first preset distance from the target image acquisition module, then the attic is identified as the other focusing target.

[0109] This method allows for the selection of a focus target from candidate focus objects based on the distance between the candidate focus object and the target image acquisition module. Furthermore, it determines the potential focus target for M image acquisition modules based on their structural attribute parameters. This reduces the difficulty of focusing for the M image acquisition modules and further improves the image quality after acquisition.

[0110] In another embodiment, step S1042, determining the focus candidates within a first preset distance from a plurality of focus candidate targets as the same focus target, includes:

[0111] Determine the third distance between each focus candidate. If there is a focus candidate whose third distance is within the second preset distance and whose second distance is within the first preset distance, then the focus candidate whose third distance is within the second preset distance and whose second distance is within the first preset distance is determined as the same focus target.

[0112] After determining the second distance between each candidate focus object and the target image acquisition module, a third distance between each candidate focus object can also be determined. Based on the third distance between each candidate focus object and the second distance between each candidate focus object and the target image acquisition module, the focus target is determined.

[0113] In this embodiment, both the second distance between each candidate focus object and the target image acquisition module, and the third distance between each candidate focus object, are used as influencing factors in determining the focus target. This method can more accurately identify candidate focus objects that are more likely to be the same focus target from among the candidate focus objects, and identify candidate focus objects that meet both conditions as the same focus target.

[0114] The method for determining the third distance is not limited and can be implemented using appropriate algorithms.

[0115] In another embodiment, reference Figure 7 This is a flowchart illustrating a process for determining M image acquisition modules from a plurality of image acquisition modules. Step S200 involves determining M image acquisition modules from a plurality of image acquisition modules based on a first distance between N focus targets and each image acquisition module, at least one of the image features of the N focus targets, and structural attribute parameters of the image acquisition modules, including:

[0116] Step S201: Based on the distance between each focus target and each image acquisition module, and the depth of field of each image acquisition module, determine K image acquisition modules from multiple image acquisition modules; wherein, the distance between each focus target and the K image acquisition modules is within the depth of field of at least one of the K image acquisition modules; K is greater than or equal to M; K, M and N are all positive integers.

[0117] Step S202: Sort the N focus targets by area according to the area corresponding to the edge contour of each focus target, and determine the first sorting result of the N focus targets.

[0118] Step S203: Sort the field of view sizes of the K image acquisition modules according to their field of view angles, and determine the second sorting result of the K image acquisition modules.

[0119] Step S204: Based on the first sorting result and the second sorting result, determine M image acquisition modules from the K image acquisition modules to focus on and acquire images of the N focus targets respectively.

[0120] For step S201, since the depth of field of each image acquisition module may be different, and the distance between each focus target and each image acquisition module is also different, it is necessary to determine which image acquisition modules can focus on each focus target before the image acquisition modules focus on each focus target. That is, which image acquisition modules can focus on each focus target. In this process, it is also determined which focus targets each of these image acquisition modules can focus on.

[0121] An image acquisition module capable of focusing on a target refers to an image acquisition module whose distance from the target is within the depth of field of the image acquisition module.

[0122] For example, the focus targets include focus target A, focus target B, and focus target C, and the multiple image acquisition modules include image acquisition module 1, image acquisition module 2, image acquisition module 3, image acquisition module 4, and image acquisition module 5. Based on the distance between each focus target and each image acquisition module, and the depth of field of each image acquisition module, image acquisition modules 1 and 2 are determined to be able to focus on focus target A. The distance between focus target A and image acquisition module 1 is within the depth of field of image acquisition module 1, and the distance between focus target A and image acquisition module 2 is within the depth of field of image acquisition module 2. Similarly, image acquisition modules 1 and 3 are able to focus on focus target B. The distance between focus target B and image acquisition module 1 is within the depth of field of image acquisition module 1, and the distance between focus target B and image acquisition module 3 is within the depth of field of image acquisition module 3. The image acquisition modules capable of focusing on the target C are image acquisition module 1, image acquisition module 2, and image acquisition module 3. The distance between the target C and image acquisition module 1 is within the depth of field of image acquisition module 1, the distance between the target C and image acquisition module 2 is within the depth of field of image acquisition module 2, and the distance between the target C and image acquisition module 3 is within the depth of field of image acquisition module 3.

[0123] The distances between focus targets A, B, and C and image acquisition module 4 are all outside the depth of field of image acquisition module 4, and the distances between focus targets A, B, and C and image acquisition module 5 are also outside the depth of field of image acquisition module 5. Therefore, image acquisition modules 1, 2, and 3 can be identified as the K image acquisition modules from the five image acquisition modules. That is, the K image modules are image acquisition modules 1, 2, and 3, where K is 3.

[0124] For example, the focus target also includes focus target D. The distance between focus target D and image acquisition module 1 is not within the depth of field of image acquisition module 1, the distance between focus target D and image acquisition module 2 is not within the depth of field of image acquisition module 2, and the distance between focus target D and image acquisition module 3 is not within the depth of field of image acquisition module 3. Since image acquisition modules 1, 2, and 3 cannot focus on focus target D, no focusing or image acquisition is performed on focus target D.

[0125] Similarly, it can be determined which targets each image acquisition module can focus on. For example, image acquisition module 1 can focus on target A, target B, and target C; image acquisition module 2 can focus on target A and target C; and image acquisition module 3 can focus on target B and target C.

[0126] In step S201, K image acquisition modules can be determined from multiple image acquisition modules. Each of these K image acquisition modules is capable of focusing on at least one of the N focus targets. That is, the distance between each focus target and each of the K image acquisition modules is at least within the depth of field of one of the K image acquisition modules. This ensures that at least one of the K image acquisition modules can focus on and acquire images of at least one focus target.

[0127] The goal of this step is to pre-select K image acquisition modules from multiple image acquisition modules. These K image acquisition modules can all focus on N focus targets. This reduces the possibility of identifying an image acquisition module that cannot focus on N focus targets, thereby reducing the possibility that the distance between the focus target and the image acquisition module is not within the depth of field of the image acquisition module, which would prevent the image acquisition module from clearly focusing on the focus target. This helps to improve the sharpness and imaging quality of the first image.

[0128] In step S202, in this embodiment, the image feature is the edge contour. After determining the focus target, the edge contour of the focus target can be further determined, and the area can be further determined based on the edge contour. The method for determining the edge contour of the focus target is not limited here; any method that can determine the edge contour of the focus target after it has been determined is acceptable. For example, edge detection algorithms, etc. Other methods will not be listed here.

[0129] The area corresponding to the edge contour of the focused target can be the area of ​​the connected region formed by connecting the edge contours.

[0130] The area corresponding to the edge contour of the focus target can also be determined by the circumscribed frame of the focus target's edge contour, such as a circumscribed rectangle or a circumscribed circle. The area of ​​the focus target is determined based on this circumscribed frame. Here, the area of ​​the circumscribed frame of the focus target is used as the reference area for sorting the areas of the focus targets. This method allows for a better comparison of the size of each focus target, as it directly compares the areas, simplifying the complexity of determining the size of each focus target and comparing the sizes, reducing the amount of computation, and without affecting the comparison results.

[0131] The area corresponding to the edge contour of each focus target is sorted by size, and the first sorting result is determined based on this area sorting. For example, the area corresponding to focus target A, the area corresponding to focus target B, and the area corresponding to focus target C decrease in that order. The first sorting result can be sorted in descending order of area.

[0132] For step S203, since different image acquisition modules have different field of view angles, the viewing angle range of the images that each image acquisition module can acquire is also different. This embodiment determines the second sorting result of the K image acquisition modules by sorting the field of view angles of the K image acquisition modules that can focus on each focus target.

[0133] The sorting order of the first sorting result corresponds to the sorting order of the second sorting result. The second sorting result can be sorted in descending order of field of view. For example, the field of view of image acquisition module 1, the field of view of image acquisition module 2, and the field of view of image acquisition module 3 decrease in sequence.

[0134] By sorting the field of view, the image acquisition module with a relatively large field of view can focus on and acquire images of the target within a wider field of view. This reduces the problem that when the field of view is small, the target is not within the image acquisition field of view of the image acquisition module, which would prevent the image acquisition module from focusing on and acquiring images of the target.

[0135] In step S204, based on the area of ​​the target's edge contour and the field of view of the image acquisition module, this step determines the image acquisition module that best matches the target from among the K image acquisition modules to focus on and acquire the image. The larger the area of ​​the target's edge contour, the larger the field of view of the matching image acquisition module. This ensures that larger targets can be focused on and have their images fully acquired.

[0136] For example, focusing on target A and target B, from K image acquisition modules, image acquisition module 1 is selected for focusing and acquiring images of target A, and image acquisition module 3 is selected for focusing and acquiring images of target B. Then, the M image acquisition modules are image acquisition module 1 and image acquisition module 3, and M is 2. In this embodiment, M is less than or equal to K.

[0137] In another embodiment, reference Figure 8 This is a flowchart illustrating another method for determining the first sorting result. S202, sorting the N focus targets by area based on the area corresponding to the edge contours of each focus target, can include:

[0138] Step S2021: Determine the sorting of the area sizes corresponding to the edge contours of each focused target.

[0139] In this embodiment, the image feature of the focus target is the edge contour. After determining the edge contour of the focus target, the area corresponding to the edge contour can be determined, and the area size can be sorted to obtain the sorting result of the area size.

[0140] Step S2022: Determine that the area corresponding to the edge contours of at least two focused targets is the same size.

[0141] When sorting the area of ​​the edge contours of the focus targets, it is necessary to determine whether there are two focus targets with the same area corresponding to their edge contours. The goal of this step is to identify focus targets with the same area corresponding to their edge contours, and then sort these focus targets with the same area corresponding to their edge contours.

[0142] Step S2023: Sort the distances between at least two focus targets of the same area and the center positions of the N focus targets in the image, and obtain the third sorting result.

[0143] In this embodiment, if it is determined that at least two focus targets have the same area corresponding to their edge contours, then the focus targets are sorted according to their distances to the center positions of the N focus targets in the image, resulting in a third sorting result. This step ensures that when sorting by area, focus targets with the same area are sorted. The third sorting result can be a sort of multiple focus targets with the same area corresponding to their edge contours.

[0144] For example, the third sorting result can be a sorting by distance from smallest to largest.

[0145] Step S2024: Determine the first sorting result based on the area size sorting and the third sorting result.

[0146] After obtaining the third sorting result, the first sorting result is further determined based on the area size and the third sorting result. This allows for the sorting of N focus targets according to the area of ​​their edge contours, resulting in the first sorting result. This first sorting result includes the sorting of focus targets with the same area and those with different areas.

[0147] For example, if the area corresponding to focus target A is the same as the area corresponding to focus target B, then the distances between focus target A and focus target B and the center positions of the N focus targets in the image are determined. These distances are then sorted in ascending order. For example, if the distance between focus target A and the center positions of the N focus targets in the image is less than the distance between focus target B and the center positions of the N focus targets in the image, then focus target A is ranked higher than focus target B. Finally, the sorted focus targets A and B are sorted with the other focus targets to obtain the first ranking result.

[0148] In another embodiment, the third sorting result can also be sorted from largest to smallest according to the distance.

[0149] In another embodiment, reference Figure 9 This is a flowchart illustrating the process of determining M image acquisition modules for focusing and acquiring images of N focus targets. Step S204 involves determining M image acquisition modules for focusing and acquiring images of N focus targets from K image acquisition modules based on a first sorting result and a second sorting result, including:

[0150] Step S2041: Based on the first sorting result, determine L image acquisition modules from K image acquisition modules that match the P-th focus target; wherein, the distance between the P-th focus target and the L image acquisition modules is within the depth of field of the L image acquisition modules; the P-th focus target is preferentially matched with the L image acquisition modules over the (P+1)-th focus target; L is less than or equal to K.

[0151] Step S2042: Based on the second sorting result, determine the image acquisition module that will focus on and acquire images of the P-th focus target from the L image acquisition modules; where P and L are both positive integers, and P is any value from 1 to N; among the L image acquisition modules, the image acquisition module ranked earlier will focus on and acquire images of the P-th focus target before the image acquisition module ranked later.

[0152] For step S2041, the idea is as follows: In the first sorting result, determine the image acquisition module that matches the focus target corresponding to each sorting position, that is, determine the image acquisition module that can perform image acquisition on the focus target corresponding to each sorting position. When determining the image acquisition module for each focus model, it is always selected from the image acquisition modules that can perform focusing and image acquisition on the focus target.

[0153] In the first ranking result, the priority of determining the image acquisition module that matches the highest-ranked focus target takes precedence over the priority of determining the image acquisition module that matches the lowest-ranked focus target. For example, the image acquisition module that matches the first-ranked focus target is determined first, then the image acquisition module that matches the second-ranked focus target is determined, and so on.

[0154] For example, from K image acquisition modules, L image acquisition modules that match the focus target A are identified as image acquisition module 1 and image acquisition module 2.

[0155] For step S2042, based on the second sorting result, the image acquisition modules that focus on the target at each sorting position in the first sorting result are determined from the L image acquisition modules. Among the determined L image acquisition modules, the image acquisition module ranked higher has priority over the image acquisition module ranked lower in the sorting for focusing and image acquisition on the P-th focus target. For the same focus target, among the image acquisition modules that can focus and acquire images of that focus target, the image acquisition module with the larger field of view has higher priority, and the image acquisition module with the largest field of view is preferentially selected as the image acquisition module for focusing and image acquisition on that focus target.

[0156] For example, the areas corresponding to focus target A, focus target B, and focus target C decrease sequentially, as do the field of view angles of image acquisition module 1, image acquisition module 2, and image acquisition module 3. Based on the first sorting result, L image acquisition modules matching focus target A are preferentially determined; that is, image acquisition modules capable of focusing on and acquiring images of focus target A include image acquisition module 1 and image acquisition module 2. Then, based on the second sorting result, from the L image acquisition modules (i.e., image acquisition module 1 and image acquisition module 2), image acquisition module 1 is determined to be the image acquisition module for focusing on and acquiring images of focus target A. This process continues.

[0157] In another embodiment, reference Figure 10 This is a flowchart illustrating another image processing method. Step S2042, based on the second sorting result, determines the image acquisition module from the L image acquisition modules that will focus on and acquire the image of the P-th focus target. This may include:

[0158] Step S20421: From the L image acquisition modules, determine R image acquisition modules, where the R image acquisition modules are: image acquisition modules other than the image acquisition modules that have been determined to focus on and acquire images of the focus target from the L image acquisition modules; R is a positive integer greater than or equal to zero and less than or equal to L.

[0159] When determining the image acquisition module to focus on and acquire images of a target at a certain position in the first sorting result, for example, if the sorting position is P, then from the L image acquisition modules determined to focus on and acquire images of the P-position target, R image acquisition modules that have not yet been determined to focus on and acquire images of targets at other sorting positions are selected. The L image acquisition modules that match the P-position target include these R image acquisition modules. The image acquisition modules other than the R image acquisition modules in the L image acquisition modules are those that have been determined to focus on and acquire images of other targets in the L image acquisition modules, that is, the image acquisition modules corresponding to the targets before the P-position target in the L image acquisition modules (including targets from position 1 to position P-1). In this case, these R image acquisition modules are the image acquisition modules in the L image acquisition modules other than those that have been determined to focus on and acquire images of other targets in the L image acquisition modules.

[0160] Step S20422: Based on the second sorting result, determine the image acquisition module that will focus on and acquire images of the P-th focusing target from the R image acquisition modules.

[0161] After determining the R image acquisition modules from the L image acquisition modules that match the P-th position, according to the second sorting result, the image acquisition module that will focus on and acquire images of the P-th focusing target is determined from the R image acquisition modules. Priority is given to determining the image acquisition module with the larger field of view to focus on and acquire images of the P-th focusing target.

[0162] For example, image acquisition module 1 can focus on focus target A, focus target B, and focus target C; image acquisition module 2 can focus on focus target A and focus target C; and image acquisition module 3 can focus on focus target B and focus target C. The area corresponding to the edge contour of focus target A, the area corresponding to the edge contour of focus target B, and the area corresponding to the edge contour of focus target C decreases sequentially, resulting in a first ranking of A, B, and C. In this case, image acquisition modules 1 and 3 can simultaneously focus on and acquire images of the first-ranked focus target A and the third-ranked focus target C.

[0163] Since the field of view of image acquisition module 1 is greater than that of image acquisition module 3, image acquisition module 1 is prioritized for focusing and image acquisition on target A. Then, based on the second sorting result, the remaining image acquisition modules are selected to focus and acquire images of target C. The remaining image acquisition modules are image acquisition modules 3, and therefore, based on the second sorting result, image acquisition module 3 is selected to focus and acquire images of target C.

[0164] In another embodiment, M image acquisition modules can be determined based on any one of the color features, texture features, shape features, or spatial relationship features of the focused target, combined with the first distance and the structural attribute parameters of the image acquisition modules.

[0165] For example, based on the first distance, the color characteristics of the focus target, and the structural attribute parameters of the image acquisition modules, M image acquisition modules are determined. Based on the distance between each focus target and each image acquisition module, and the depth of field of each image acquisition module, K image acquisition modules are determined from the multiple image acquisition modules. Then, based on the color characteristics of each focus target, a color sort is performed to determine the first sorting result. Finally, based on the preset order of the K image acquisition modules, M image acquisition modules are determined to perform focusing and image acquisition for each focus target respectively.

[0166] For example, based on the first distance, the texture features of the focus target, and the structural attribute parameters of the image acquisition modules, M image acquisition modules are determined. Based on the distance between each focus target and each image acquisition module, and the depth of field of each image acquisition module, K image acquisition modules are determined from the multiple image acquisition modules. Then, based on the texture features of each focus target, the texture features are sorted by coarseness and / or density to determine the first sorting result. Next, based on the focusing distance of the K image acquisition modules, a second sorting result is determined. Based on the first and second sorting results, M image acquisition modules are determined to focus and acquire images for each focus target respectively. Color features can be extracted using a color feature extraction algorithm.

[0167] For example, an image acquisition module with a longer focusing distance is used to focus on and acquire images of targets with finer or denser texture features, while an image acquisition module with a shorter focusing distance is used to focus on and acquire images of targets with coarser or sparser texture features.

[0168] For example, based on the first distance, the shape features of the focus target, and the structural attribute parameters of the image acquisition modules, M image acquisition modules are determined. Based on the distance between each focus target and each image acquisition module, and the depth of field of each image acquisition module, K image acquisition modules are determined from the multiple image acquisition modules. Then, based on the shape features of each focus target, shape features are sorted to determine the first sorting result, which can be based on a first preset order. Then, based on a second preset order of the K image acquisition modules, which can be based on factors such as the field of view, M image acquisition modules are determined to focus on and acquire images from each focus target. Shape features can be extracted using a shape feature extraction algorithm.

[0169] For example, the image acquisition module with a larger field of view in the second preset sequence is selected to focus on and acquire images of a circular focus target.

[0170] For example, based on the first distance, the spatial relationship characteristics of the focus target, and the structural attribute parameters of the image acquisition module, M image acquisition modules are determined. Based on the distance between each focus target and each image acquisition module, and the depth of field of each image acquisition module, K image acquisition modules are selected from the multiple image acquisition modules. Then, based on the spatial relationship characteristics of each focus target, focus targets with a preset spatial relationship are determined to use the same image acquisition module for focusing and image acquisition. Spatial relationship characteristics can be extracted using a spatial relationship feature extraction algorithm.

[0171] In another embodiment, reference Figure 11 This is a schematic diagram of the structure of an image processing device, which may include: a focus target determination module 1, an image acquisition module determination module 2, a control module 3, and a fusion module 4.

[0172] The focus target determination module 1 is used to determine N focus targets within the framing area.

[0173] Image acquisition module determination module 2 is used to determine M image acquisition modules among the plurality of image acquisition modules based on the first distance between the N focus targets and each of the image acquisition modules, at least one of the image features of the N focus targets, and the structural attribute parameters of the image acquisition modules; M is less than or equal to N.

[0174] Control module 3 is used to control the M image acquisition modules to focus and acquire images of the N focus targets respectively, wherein each of the M image acquisition modules corresponds to a different focus target.

[0175] The fusion module 4 is used to fuse the first image acquired by each of the M image acquisition modules to generate a second image.

[0176] In another embodiment, the image acquisition module determining module 2 may include:

[0177] The first image acquisition module determination submodule is used to determine K image acquisition modules from a plurality of image acquisition modules based on the distance between each focus target and each image acquisition module, and the depth of field of each image acquisition module; wherein, the distance between each focus target and the K image acquisition modules is within the depth of field of at least one of the K image acquisition modules; K is greater than or equal to M; K, M and N are all positive integers.

[0178] The first sorting result determination submodule is used to sort the N focus targets by area based on the area corresponding to the edge contour of each focus target, and determine the first sorting result of the N focus targets.

[0179] The second sorting result determination submodule is used to sort the field of view sizes of the K image acquisition modules according to the field of view angles, and determine the second sorting result of the K image acquisition modules.

[0180] The second image acquisition module determination submodule is used to determine, from the K image acquisition modules, the M image acquisition modules that focus on and acquire images of the N focus targets respectively, based on the first sorting result and the second sorting result.

[0181] In another embodiment, the first sorting result determining submodule may include:

[0182] An area sorting unit is used to determine the sorting of the area sizes corresponding to the edge contours of each of the focused targets.

[0183] A determining unit is used to determine that at least two of the said edge contours correspond to areas of the same size.

[0184] The third sorting result determination unit is used to sort the objects based on the distance between at least two of the focusing targets with the same area and the center position of the N focusing targets in the image, and obtain the third sorting result.

[0185] The first sorting result determination unit is used to determine the first sorting result based on the area size sorting and the third sorting result.

[0186] In another embodiment, the second image acquisition module determining submodule may include:

[0187] The first image acquisition module determining unit is configured to determine, based on the first sorting result, L image acquisition modules from the K image acquisition modules that match the P-th focusing target; wherein the distance between the P-th focusing target and the L image acquisition modules is within the depth of field of the L image acquisition modules; the P-th focusing target is preferentially matched with the L image acquisition modules over the (P+1)-th focusing target; and L is less than or equal to K.

[0188] The second image acquisition module determining unit is used to determine, from the L image acquisition modules, the image acquisition module that performs focusing and image acquisition on the P-th focusing target from the second sorting result; wherein, P and L are both positive integers, and P is any value from 1 to N; among the L image acquisition modules, the image acquisition module ranked earlier is given priority over the image acquisition module ranked later in the sorting for focusing and image acquisition on the P-th focusing target.

[0189] In another embodiment, the second image acquisition module determining unit may include:

[0190] The first image acquisition module determining subunit is used to determine R image acquisition modules from the L image acquisition modules, wherein the R image acquisition modules are: image acquisition modules other than the image acquisition modules that have been determined to focus on and acquire images of the focus target from the L image acquisition modules; R is a positive integer greater than or equal to zero and less than or equal to L;

[0191] The second image acquisition module determination subunit determines, based on the second sorting result, the image acquisition module that performs focusing and image acquisition on the P-th focusing target from the R image acquisition modules.

[0192] In another embodiment, the focus target determination module 1 may include:

[0193] The preview image acquisition unit is used to acquire a preview image of the target image acquisition module.

[0194] The focus candidate determination unit determines multiple focus candidates based on the preview image.

[0195] A depth information acquisition unit is used to acquire the depth information of the preview image.

[0196] The focus target determination unit is used to determine the N focus targets from the plurality of focus candidate objects based on the depth information.

[0197] In another embodiment, the focus target determination unit may include:

[0198] The second distance determination subunit is used to determine the second distance between each of the focus candidate objects and the target image acquisition module based on the depth information.

[0199] The focus target determination subunit is used to determine the focus candidates within the second distance and the first preset distance from the plurality of focus candidate objects as the same focus target.

[0200] In another embodiment, the focus candidate determination unit includes:

[0201] The identification subunit is used to identify the foreground and background areas in the preview image;

[0202] The focus candidate object determination subunit is used to determine the plurality of focus candidate objects based on the foreground region.

[0203] In another embodiment, an electronic device is also provided, which may include:

[0204] A processor and a memory for storing executable instructions capable of running on the processor, wherein:

[0205] When the processor runs the executable instructions, the executable instructions perform the method described in any of the above embodiments.

[0206] In another embodiment, a non-transitory computer-readable storage medium is also provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the method described in any of the above embodiments.

[0207] It should be noted that the terms "first" and "second" in the embodiments of this disclosure are for ease of description and distinction only, and have no other specific meaning.

[0208] In another embodiment, an example processing method is also provided, including the following processing steps:

[0209] Step 1: Identify N focus targets within the framing area;

[0210] Step 2: Based on the first distance between the N focus targets and each image acquisition module, at least one of the image features of the N focus targets, and the structural attribute parameters of the image acquisition modules, determine M image acquisition modules among the multiple image acquisition modules.

[0211] Step 3: Control M image acquisition modules to focus and acquire images of N focus targets respectively. Each of the M image acquisition modules corresponds to a different focus target.

[0212] Step four: merge the first images acquired by each of the M image acquisition modules to generate the second image.

[0213] Step 1: Identify N focus targets within the viewfinder area, including:

[0214] 1) Obtain a preview image of the target image acquisition module.

[0215] In this embodiment, the target image acquisition module can be determined according to the configuration of the terminal device. For example, the main camera module in the terminal device can be a wide-angle camera or a telephoto camera.

[0216] 2) Based on the preview image, identify multiple potential focus targets.

[0217] For example, the foreground and background regions in the preview image are identified, and the preview image is processed using a saliency map algorithm to obtain a saliency map. The saliency map includes black and white areas; here, the white areas are identified as the foreground, and the black areas as the background. Based on the foreground areas, multiple potential focus objects are identified; here, the white areas are identified as potential focus objects.

[0218] For example, after identifying multiple focus candidates, the process may further include: processing the candidate focus candidates, such as processing the foreground region through a closing operation. The foreground region after the closing operation is determined as a focus candidate. By performing a closing operation on the foreground region, foreground regions that meet the closing operation processing conditions can be further filtered and / or connected to obtain more accurate focus candidates for subsequent focusing.

[0219] 3) Obtain the depth information of the preview image.

[0220] The depth information of the target image acquisition module's preview image can be obtained by triangulation using preview images from the target image acquisition module and another image acquisition module. Alternatively, the depth information of the target image acquisition module's preview image can be obtained using a Time-of-Flight (TOF) structure.

[0221] 4) Based on the depth information, determine N focus targets from multiple focus candidate objects.

[0222] Based on depth information, a second distance is determined between each potential focus object and the target image acquisition module. Among multiple potential focus objects, those whose second distance is within a first preset distance are identified as the same focus target. For example, foreground regions whose second distance to the target image acquisition module is within the first preset distance are identified as the same focus target, resulting in N focus targets.

[0223] Step two includes:

[0224] 5) Based on the distance between each focus target and each image acquisition module, and the depth of field of each image acquisition module, determine K image acquisition modules from multiple image acquisition modules; wherein, the distance between each focus target and the K image acquisition modules is within the depth of field of at least one of the K image acquisition modules.

[0225] 6) Sort the N focus targets by area based on the area corresponding to the edge contour of each focus target, and determine the first sorting result of the N focus targets.

[0226] This embodiment uses the edge contour of the focused target as image features. The edge contour of the focused target is determined using an edge detection algorithm, and then the area corresponding to the edge contour can be determined by identifying the circumscribed rectangle or circumscribed circle, etc. The areas are then sorted from largest to smallest to obtain the first sorting result.

[0227] 7) Sort the field of view of the K image acquisition modules according to their field of view size to determine the second sorting result of the K image acquisition modules.

[0228] The second sorting result is obtained by sorting the fields of view from largest to smallest.

[0229] 8) Based on the first and second sorting results, determine M image acquisition modules from the K image acquisition modules to focus on and acquire images of the N focus targets respectively.

[0230] Based on the first ranking result, the image acquisition module that matches the highest-ranked focus target is prioritized, that is, the image acquisition module that matches the focus target with a relatively large area is prioritized, and the larger the area, the higher the priority. Then, based on the second ranking result, the image acquisition module with the larger field of view is prioritized to focus on and acquire images of the focus target at a certain position in the first ranking result.

[0231] In another embodiment, if the area corresponding to the edge contours of multiple focus targets is the same, then the focus targets are sorted by distance based on the distance between these focus targets and the center position of the N focus targets in the image, thereby determining the sorting result of these focus targets, and then determining the first sorting result of the N focus targets.

[0232] In another embodiment, when focusing between individual focus targets, the focus is performed on the center point of the focus target.

[0233] In another embodiment, the image acquisition module for focusing and acquiring images of the target can be determined based on the relationship between the data of the target and the number of image acquisition modules. When N is less than or equal to the number of image acquisition modules in the terminal device, the image acquisition module matching the target with a relatively large area is preferentially determined according to the first sorting result, and then the image acquisition module with a larger field of view is preferentially determined to focus on the target according to the second sorting result.

[0234] When N is greater than the number of image acquisition modules in the terminal device, based on the relationship between the area corresponding to the edge contour of the focus target and a preset area, if the area corresponding to the edge contour of the focus target is greater than the preset area, the image acquisition module matching the focus target with the larger area is prioritized according to the first sorting result. If the areas are the same, the image acquisition module matching the focus model that is relatively closer to the center of the image where the focus target is located is prioritized. If each image acquisition module has been determined to focus and acquire images for different focus targets, the determination of the image acquisition modules is completed.

[0235] Step two yields the first images of the N focused targets acquired by the M image acquisition modules.

[0236] After completing step three, step four involves fusing the first images acquired by each of the M image acquisition modules to generate the second image.

[0237] Finally, based on the preview image of the target image acquisition module, the first images of each image acquisition module are fused together to obtain the second image.

[0238] Figure 12 This is a block diagram illustrating a terminal device according to an exemplary embodiment. For example, the terminal device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0239] Reference Figure 12 The terminal device may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0240] Processing component 802 typically controls the overall operation of the terminal device, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0241] Memory 804 is configured to store various types of data to support operation on the terminal device. Examples of this data include instructions for any application or method operating on the terminal device, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0242] Power component 806 provides power to various components of the terminal device. Power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0243] Multimedia component 808 includes a screen that provides an output interface between a terminal device and a user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the terminal device is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0244] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when the terminal device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0245] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0246] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of the terminal device. For example, sensor assembly 814 can detect the on / off state of the terminal device, the relative positioning of components such as the display and keypad of the terminal device, changes in the position of the terminal device or a component of the terminal device, the presence or absence of user contact with the terminal device, the orientation or acceleration / deceleration of the terminal device, and temperature changes of the terminal device. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0247] Communication component 816 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0248] In an exemplary embodiment, the terminal device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0249] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0250] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An image processing method applied to an electronic device, the electronic device having multiple image acquisition modules, characterized in that, include: Identify N focus targets within the framing area; where N is a positive integer greater than or equal to 2; Based on the first distance between the N focusing targets and each of the image acquisition modules, the image features of the N focusing targets, and the structural attribute parameters of the image acquisition modules, M image acquisition modules are determined among the plurality of image acquisition modules; M is a positive integer less than or equal to N; The M image acquisition modules are controlled to focus and acquire images of the N focus targets respectively, wherein each of the M image acquisition modules corresponds to a different focus target; The first image acquired by each of the M image acquisition modules is merged to generate the second image.

2. The method according to claim 1, characterized in that, The step of determining M image acquisition modules from the plurality of image acquisition modules based on the first distance between the N focus targets and each of the image acquisition modules, the image features of the N focus targets, and the structural attribute parameters of the image acquisition modules includes: Based on the first distance between each of the focusing targets and each of the image acquisition modules, and the depth of field of each of the image acquisition modules, K image acquisition modules are determined from the plurality of image acquisition modules; wherein, the distance between each of the focusing targets and the K image acquisition modules is within the depth of field of at least one of the K image acquisition modules; K is greater than or equal to M; K is a positive integer; The N focusing targets are sorted by area based on the area corresponding to the edge contour of each focusing target, and the first sorting result of the N focusing targets is determined. The field of view of the K image acquisition modules is sorted by size to determine the second sorting result of the K image acquisition modules; Based on the first sorting result and the second sorting result, M image acquisition modules are determined from the K image acquisition modules to focus on and acquire images of the N focus targets respectively.

3. The method according to claim 2, characterized in that, The step of sorting the N focus targets by area based on the area corresponding to the edge contour of each focus target to determine the first sorting result includes: Determine the order of the area sizes corresponding to the edge contours of each of the focused targets; Determine that at least two of the said edge contours correspond to the same area size; Based on the distance between at least two of the focus targets with the same area and the center position of the N focus targets in the image, the distances are sorted to obtain a third sorting result; The first sorting result is determined based on the area size sorting and the third sorting result.

4. The method according to claim 2 or 3, characterized in that, The step of determining, from the K image acquisition modules, the M image acquisition modules for focusing and acquiring images of the N focus targets respectively, based on the first sorting result and the second sorting result includes: Based on the first sorting result, L image acquisition modules that match the P-th focus target are determined from the K image acquisition modules; wherein the distance between the P-th focus target and the L image acquisition modules is within the depth of field of the L image acquisition modules; the P-th focus target is preferentially matched with the L image acquisition modules over the (P+1)-th focus target; L is less than or equal to K; Based on the second sorting result, from the L image acquisition modules, the image acquisition module that performs focusing and image acquisition on the P-th focusing target is determined; where P and L are both positive integers, and P is any value from 1 to N; among the L image acquisition modules, the image acquisition module ranked earlier performs focusing and image acquisition on the P-th focusing target with priority over the image acquisition module ranked later.

5. The method according to claim 4, characterized in that, The step of determining the image acquisition module for focusing and acquiring images of the P-th focus target from the L image acquisition modules according to the second sorting result includes: From the L image acquisition modules, R image acquisition modules are determined, wherein the R image acquisition modules are: image acquisition modules other than the image acquisition modules that have been determined to focus on and acquire images of the focus target from the L image acquisition modules; R is a positive integer greater than zero and less than or equal to L; Based on the second sorting result, the image acquisition module that performs focusing and image acquisition on the P-th focusing target is determined from the R image acquisition modules.

6. The method according to claim 1, characterized in that, The determination of N focus targets within the framing area includes: Obtain a preview image of the target image acquisition module; Based on the preview image, multiple focus candidates were identified; Obtain the depth information of the preview image; Based on the depth information, the N focus targets are determined from the plurality of focus candidate objects.

7. The method according to claim 6, characterized in that, The step of determining the N focus targets from the plurality of focus candidate objects based on the depth information includes: Based on the depth information, a second distance is determined between each of the focus candidate objects and the target image acquisition module; Among the plurality of focus candidates, those focus candidates that are within the second distance of the first preset distance are determined to be the same focus target.

8. The method according to claim 7, characterized in that, The step of determining the focus candidates within the second distance of the plurality of focus candidates as the same focus target includes: Determine the third distance between each of the aforementioned focus candidates; Among the plurality of focus candidates, those focus candidates whose third distance is within the second preset distance and whose second distance is within the first preset distance are determined to be the same focus target.

9. The method according to claim 6, characterized in that, The step of determining multiple focus candidates based on the preview image includes: Identify the foreground and background areas in the preview image; Based on the foreground region, the plurality of focus candidates are determined.

10. An image processing apparatus, applied to an electronic device, the electronic device having multiple image acquisition modules, characterized in that, include: The focus target determination module is used to determine N focus targets within the framing area; where N is a positive integer greater than or equal to 2. An image acquisition module determination module is used to determine M image acquisition modules among the plurality of image acquisition modules based on the first distance between the N focus targets and each of the image acquisition modules, the image features of the N focus targets, and the structural attribute parameters of the image acquisition modules. M is a positive integer less than or equal to N; A control module is used to control the M image acquisition modules to focus and acquire images of the N focus targets respectively, wherein each of the M image acquisition modules corresponds to a different focus target; The fusion module is used to fuse the first images acquired by each of the M image acquisition modules to generate a second image.

11. An electronic device, characterized in that, include: A processor and a memory for storing executable instructions capable of running on the processor, wherein: When the processor is used to run the executable instructions, the executable instructions perform the method described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the method described in any one of claims 1 to 9.

Images