A method and device for enhancing optimal snapshot effect

Abstract

The present invention relates to a method and device for enhancing optimal capture effects. The method primarily includes: recording the target image quality score of a target during target tracking; recording the target area with the highest target image quality score during target tracking, if the target disappears or times out; statistically analyzing the target tracking process over a period of time or for a certain number of targets, and updating the optimal capture area; calculating the average target brightness for targets in the optimal capture area, determining whether exposure adjustment is necessary based on the average target brightness, and performing exposure adjustment if necessary. The present invention can focus on the optimal capture area, preventing targets in non-optimal capture areas from affecting exposure quality.

Description

Technical Field

[0001] The present invention relates to the technical field of security monitoring, and in particular to a method and device for enhancing an optimal snapshot effect. Background Art

[0002] Traditional video surveillance captures the entire image. In most scenes with front lighting, back lighting, or uneven lighting, it is difficult to obtain target snapshots with satisfactory brightness and contrast.

[0003] To alleviate this problem, researchers have proposed various local exposure methods, which can improve the quality of target capture to a certain extent. These methods primarily focus on exposing a specific target within the frame. Typically, local exposure methods average the brightness of all detected targets. If the average is greater than a threshold, the exposure is reduced; if it is less than the threshold, the exposure is increased.

[0004] However, this technology has the following problems: when there are multiple targets in the picture at the same time, the exposure of multiple targets is often averaged. In this way, when the brightness distribution in the environmental area is uneven, targets in different brightness areas will affect each other, resulting in the average image quality of the target capture to deteriorate.

[0005] In addition, when the camera direction or shooting area changes, the optimal capture area often changes. At this time, the user is often required to manually set the optimal capture area, which is very troublesome.

[0006] In view of this, how to overcome the defects of the existing technology and enhance the optimal capture effect is a difficult problem to be solved in this technical field. Summary of the Invention

[0007] In view of the defects or improvement needs in the existing technology: (1) When there are multiple targets in the picture at the same time, if some targets are in the area with suitable ambient brightness and some targets are in the area with unsuitable brightness, the average exposure of all targets will cause the targets with unsuitable ambient brightness to affect the targets with suitable ambient brightness, that is, the quality of the captured images of the targets in the area with suitable ambient brightness will deteriorate. (2) When the camera direction or shooting area changes, the optimal capture area often changes. In this case, the user is often required to manually set the optimal capture area, which is very troublesome. The present invention uses a target tracking and target image quality evaluation algorithm to record the target image quality score during the target tracking process, and records the target area with the maximum target image quality score during the target tracking process when the target disappears or times out. Then, the target tracking process within a period of time or a certain number of targets is counted, and the optimal capture area is updated through the area update algorithm. Finally, the detected targets are filtered, the target average brightness is calculated for the targets in this area, and it is determined whether the target average brightness needs to be adjusted for exposure. If necessary, the exposure is adjusted based on the target average brightness in the optimal capture area. Through the improvements of the present invention, it is possible to focus on the optimal capture area and avoid the impact of targets in non-optimal capture areas on exposure quality. In addition, since the present invention can update the optimal capture area according to the prescribed strategy, when the user changes the camera position or the camera shooting environment changes, if the image difference between the two target detections is less than the threshold, it indicates that the image environment has not changed much, and the original area will not be cleared. When the target is detected, it will be updated on the original optimal capture area. If the image difference between the two detections is greater than the threshold, it indicates that the image environment has changed significantly, and the original area will be cleared and the optimal capture area will be regenerated. When the number of re-detection targets reaches a predetermined number, the newly generated optimal capture area is the optimal capture area for the new scene.

[0008] The embodiment of the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a method for enhancing an optimal snapshot effect, comprising:

[0010] During the target tracking process, the target image quality score of the target is recorded. When the target disappears or times out, the target area with the maximum target image quality score during the target tracking process is recorded.

[0011] Statistics the target tracking process within a period of time or a certain number of targets, and update the best capture area;

[0012] Calculate the average brightness of the target in the optimal capture area, and determine whether exposure adjustment is needed based on the target average brightness. If necessary, perform exposure adjustment.

[0013] Furthermore, obtaining the target image quality score specifically includes:

[0014] Use the target recognition algorithm to identify the target in the image and output the target area array after recognition. The target area array represents the location information of all detected targets. Each target area element in the target area array contains coordinates and width and height;

[0015] The original image is segmented by the coordinates and width and height of the target area array to obtain the target image array to represent the target images of all detected targets;

[0016] A target image quality score algorithm is used to perform target image quality scores on all target image arrays to obtain a target quality score array to represent the target image quality scores of all detected targets.

[0017] Furthermore, the target image quality score of the target is recorded during the target tracking process, and the target area with the maximum target image quality score during the target tracking process is recorded when the target disappears or times out. Specifically, the target image quality score of the target is recorded during the target tracking process.

[0018] The target is tracked using a target tracking algorithm. The tracking process repeats the acquisition of the target area array, the target image array, and the target quality score array, while simultaneously capturing the target.

[0019] If the target is captured for the first time, the target area and target image quality score of this capture are recorded; if the target is not captured for the first time, it is determined whether the target image quality score of the target in this tracking process is greater than the target image quality score of the previous time. If so, the target area and target image quality score of the previous time are replaced with the target area and target image quality score of the current time;

[0020] When the target tracking algorithm determines that the target disappears or the residence time exceeds the preset threshold, the target area with the highest target image quality score during the tracking process is output for subsequent optimal capture area update.

[0021] Furthermore, the target tracking process within a period of time or a certain number of targets is counted and the optimal capture area is updated, specifically including:

[0022] The images during target tracking within a certain period of time or a certain number of targets are divided into pre-set grids, and the capture order of different target areas is stored in the two-dimensional array data grid_queue. Each two-dimensional array element maintains a queue. The maximum length of the queue is the threshold TargetNum, which represents the number of captures required to form the optimal capture area.

[0023] Each time the best capture area needs to be updated, the target area with the highest target image quality score is added to the queue of the corresponding grid with the first mark, and the queue of the non-corresponding grid is added to the second mark. It is judged whether the total number of area updates targetnum is greater than the threshold TargetNum. If so, all the data at the end of the grid queue is dequeued, otherwise it is only added to the queue but not dequeued;

[0024] The grid array grid_capture is used to represent the optimal capture area. The value of each grid array element indicates whether the target area is in the optimal capture area.

[0025] Furthermore, determining whether the grid is the target area corresponding to the grid with the highest target image quality score specifically includes:

[0026] Determine the grids where the coordinates of the four corners of the target area with the highest target image quality score are located. If the four corners are in different grids, four grids will be formed, and the rectangular grid area surrounded by these four grids will be used as the corresponding grid area of ​​the target area with the highest target image quality score. If the four corners are in only two grids or only one grid, the two grids and the area between them or the single grid will be used as the corresponding grid area of ​​the target area with the highest target image quality score.

[0027] Furthermore, the grid array grid_capture is used to represent the optimal capture area, and the value of each grid array element indicates whether the target area is in the optimal capture area. Specifically, it includes:

[0028] Determine whether the total number of area updates targetnum is greater than or equal to the threshold TargetNum. If not, it means that the total number of updates is insufficient. All grids are used to represent the optimal capture area, and the first mark is assigned to each element of the grid array grid_capture. If the total number of area updates targetnum is greater than or equal to the threshold TargetNum, the optimal capture area is updated enough. The two-dimensional array data grid_queue[i][j] representing the grid queue is judged. If there is data in the queue that is not the second mark, it means that this area is within the optimal capture area. The first mark is assigned to the corresponding grid array grid_capture[i][j]. If the queue data are all the second mark, the second mark is assigned to grid_capture[i][j]. Among them, grid_capture[i][j] indicates whether the area corresponding to the area of ​​the i-th row and j-th column is the optimal capture area; grid_queue[i][j] represents the queue corresponding to the area of ​​the i-th row and j-th column.

[0029] Furthermore, it also includes filtering areas with low update times. Specifically: polling the grid array grid_capture of the best capture area, calculating the update frequency p of each grid, the larger the update frequency p, the more times the highest target image quality score is updated in this grid. If the update frequency p is less than the preset frequency threshold UpdateThd, the area corresponding to the grid is filtered out.

[0030] Furthermore, calculating the target average brightness of the target in the optimal capture area, determining whether exposure adjustment is required based on the target average brightness, and performing exposure adjustment if required specifically includes:

[0031] Output the target area array target_area[num], each target area array element contains coordinates and width and height (x, y, width, height), num represents the number of recognized targets;

[0032] Filter the target, and if the target area is not in the optimal capture area, delete the target; the method of determining whether the target area is in the optimal capture area includes: determining whether the overlap between the target area and the optimal capture area is greater than a preset overlap threshold; if not, determining that the target area is not in the optimal capture area;

[0033] The array of the filtered target area is target_area[filted_num], where filted_num represents the number of filtered targets. The average brightness of the filtered target area is then calculated.

[0034] Determine whether the average brightness of the filtered target area is within the ideal range. If not, optimize the brightness to the ideal range by adjusting the exposure time or gain.

[0035] Furthermore, the calculation of the average brightness of the filtered target area specifically includes:

[0036] Traverse the filtered target area array and first calculate the average brightness of a single target. The average brightness calculation method of a single target includes: obtaining the original image in YUV format, accumulating and averaging the Y value of each pixel in the target area to obtain the average brightness of the single target; then perform brightness averaging calculation on the average brightness of all single targets to obtain the final average brightness of the filtered target area.

[0037] Furthermore, when the camera position is changed or the camera shooting environment changes:

[0038] If the difference between the two target detection images is less than the preset threshold, it indicates that the image environment has not changed much, and the original optimal capture area will not be cleared. When the target is detected, it will be updated on the original optimal capture area. The image difference refers to the difference between the two captures.

[0039] If the difference between the images detected before and after is greater than the preset threshold, it indicates that the image environment has changed significantly. In this case, the original best capture area will be cleared and a new best capture area will be generated. When the number of re-detection targets reaches the set number, the newly generated best capture area will be the best capture area for the new scene.

[0040] On the other hand, the present invention provides an implementation device for enhancing the optimal snapshot effect, specifically: including at least one processor and a memory, at least one processor and the memory are connected through a data bus, the memory stores instructions that can be executed by at least one processor, and after the instructions are executed by the processor, they are used to complete the implementation method of enhancing the optimal snapshot effect in the first aspect.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention can focus on the optimal capture area to prevent targets in non-optimal capture areas from affecting exposure quality.

[0043] 2. Because the present invention can update the optimal capture area according to a specified strategy, when the user changes the camera position or the camera shooting environment changes, if the difference between the two target detection images is less than a threshold, indicating that the image environment has not changed much, the original area will not be cleared, and the original optimal capture area will be updated when the target is detected. If the difference between the two detected images is greater than the threshold, indicating that the image environment has changed significantly, the original area will be cleared and the optimal capture area will be regenerated. When the number of target re-detections reaches a predetermined number, the newly generated optimal capture area will become the optimal capture area for the new scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0045] Figure 1 A flowchart of a method for enhancing the optimal snapshot effect provided in Example 1 of the present invention;

[0046] Figure 2 A schematic diagram of a target image quality score acquisition process according to Embodiment 1 of the present invention;

[0047] Figure 3 This is an expanded flowchart of step 100 provided in Example 1 of the present invention;

[0048] Figure 4 This is an expanded flowchart of step 200 provided in Example 1 of the present invention;

[0049] Figure 5 A schematic diagram of a capture area update process including an optimal capture area update strategy provided in Example 1 of the present invention;

[0050] Figure 6 This is an example diagram of the snapshot area update process provided in Example 1 of the present invention;

[0051] Figure 7 This is a schematic diagram of an example of updating the snapshot area provided in Example 1 of the present invention;

[0052] Figure 8 This is an expanded flowchart of step 300 provided in Example 1 of the present invention;

[0053] Figure 9 A comparison chart of different solutions in a certain scenario provided by Example 2 of the present invention;

[0054] Figure 10 This is a schematic diagram of the structure of a device for enhancing the optimal snapshot effect provided by Example 3 of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] The present invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0057] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other, and the order of the steps can be swapped if they are logical and do not conflict. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0058] Example 1:

[0059] like Figure 1 As shown, an embodiment of the present invention provides a method for enhancing the optimal snapshot effect, which includes the following steps.

[0060] Step 100: Record the target image quality score of the target during the target tracking process, and record the target area with the maximum target image quality score during the target tracking process when the target disappears or times out.

[0061] Step 200: Count the target tracking process within a period of time or a certain number of targets, and update the optimal capture area.

[0062] Step 300: Calculate the target average brightness for the target in the optimal capture area, determine whether exposure adjustment is required based on the target average brightness, and perform exposure adjustment if necessary.

[0063] This embodiment uses a target tracking and target image quality evaluation algorithm to record the target image quality score during the target tracking process, and records the target area with the highest image quality score during the target tracking process when the target disappears or times out. Then, the target tracking process within a period of time or a certain number of targets is counted, and the optimal capture area is updated through the area update algorithm. Finally, the detected targets are filtered, and the target average brightness is calculated for the targets in this area. It is determined whether the exposure needs to be adjusted by judging the target average brightness. If necessary, the exposure is adjusted with reference to the target average brightness in the optimal capture area. Through the above steps, this embodiment can focus on the optimal capture area to prevent targets in non-optimal capture areas from affecting the exposure quality.

[0064] Specifically, refer to Figure 2 In this embodiment, the acquisition of the target image quality score may specifically include the following steps.

[0065] Step 10: Use a target recognition algorithm to identify targets in the image. After identification, the target area array is output. The target area array represents the location information of all detected targets. Each target area element in the target area array contains coordinates, width, and height. Specifically, the target recognition algorithm used in this step is not limited, such as the target detection algorithm Yolov5. After detection, the target area array target_area[num] is output. Each array element contains coordinates and width and height (x, y, width, height).

[0066] Step 11: Segment the original image using the coordinates, width, and height of the target area array to obtain a target image array, which represents the target images of all detected targets. Specifically, in this step, the image segmentation module segments the original image using the coordinates, width, and height (x, y, width, height) of the target area array target_area[num] to obtain a target image array target_image[num], which represents the images of all detected targets.

[0067] Step 12: Perform a target quality scoring algorithm on all target image arrays, generating a target quality score array representing the target image quality scores of all detected targets. Specifically, in this step, the target quality scoring algorithm is used to perform a target quality scoring algorithm on all detected target image arrays, target_image[num], resulting in a target quality score array, target_quality[num], representing the quality scores of all detected targets. The target quality scoring algorithm can be any algorithm, such as the FaceQnet face quality scoring algorithm or an image quality scoring formula.

[0068] refer to Figure 3 , step 100 of this embodiment (recording the target image quality score of the target during the target tracking process, and recording the target area when the target image quality score is the largest during the target tracking process when the target disappears or times out) can be expanded to the following steps.

[0069] Step 101: Target tracking is performed using a target tracking algorithm. The tracking process repeats the acquisition of the target region array, target image array, and target quality score array, while simultaneously capturing the target. Specifically, any target tracking algorithm can be used in this step, including existing algorithms such as Bytetrack. The tracking process repeats the scoring operations from steps 10 to 12, while simultaneously capturing the target.

[0070] Step 102: If the target is captured for the first time, the target area and target image quality score of this capture are recorded; if the target is not captured for the first time, it is determined whether the target image quality score of the target in this tracking process is greater than the target image quality score of the previous time. If so, the target area and target image quality score of the previous time are replaced with the target area and target image quality score of this time.

[0071] Step 103: When the target tracking algorithm determines that the target has disappeared or its dwell time exceeds a preset threshold, it outputs the target area with the highest target image quality score during the current tracking process for subsequent optimal capture area updates. Specifically, in this step, the preset threshold for target disappearance or dwell time can be set as needed; for example, it can be set to 1 minute. It should be noted that the target area array records the areas of multiple targets identified in an image, while the target quality score array records the scores of multiple targets identified in an image. The array does not store quality scores at multiple times, so there is no process for finding the maximum score within this array. The specific method for finding the maximum target score is to compare the command score of the current captured image with the previous one. If the score is higher, the target area and score in the array are replaced; if the score is lower, they are not replaced. Using this bubbling method, the area with the highest score is ultimately obtained.

[0072] refer to Figure 4 , step 200 of this embodiment (statisticing the target tracking process within a period of time or a certain number of targets, and updating the optimal capture area) can be expanded into the following steps.

[0073] Step 201: Divide the image during the target tracking process over a period of time or a certain number of targets into a pre-set grid, and use the two-dimensional array data grid_queue to store the capture order of different target areas. Each two-dimensional array element maintains a queue, and the maximum length of the queue is the threshold TargetNum, which represents the number of captures required to form the optimal capture area. Specifically, in this step, the image is divided into a pre-set grid (such as a 5*5 grid), with the grid width being Width and the height being Height. A two-dimensional array data grid_queue[Width][Height] is used to store the capture order of different areas.

[0074] Step 202: Each time the optimal capture area needs to be updated, the queue corresponding to the target area with the highest target image quality score is added to the first mark (the first mark can be 1, true, or other marks, distinguishable from the second mark), and the queues of non-corresponding grids are added to the second mark (the second mark can be 0, false, or other marks, distinguishable from the first mark). A determination is made as to whether the total number of area updates, targetnum, is greater than the threshold, TargetNum. If so, all grid tail data is dequeued; otherwise, only the data is added to the queue and not dequeued. Specifically, in this step, determining whether a grid corresponds to the target area with the highest target image quality score can be performed as follows: the grids where the four corners of the target area with the highest target image quality score are located are determined. If the four corners are located in different grids, four grids are formed, and the rectangular grid area enclosed by these four grids is treated as the target corresponding grid area. If the four corners of the target are located in only two grids or only in one area, the two grids and the area between them, or the single grid, are treated as the target corresponding grid area. It should be noted that the four grids are treated as a rectangle in this solution because the detection results of current deep learning-based object detection are all rectangular. When the four corners form a rectangle, the four corners cannot be in three grids, but only 1, 2, or 4 grids.

[0075] It should also be noted that the "update required" in step 202 refers to the need to update the optimal capture area after each target capture. In this embodiment, the relationship between "update" and "capture" is a mutually reinforcing one: "capture" provides "update" with the opportunity and area to update, and "update" provides "capture" with higher capture quality. Simply put, the process of calculating environmental characteristics through multiple captures improves capture quality, and updating is the process of calculating environmental characteristics.

[0076] Step 203: Use the grid array grid_capture to represent the optimal capture area. The value of each grid array element indicates whether the target area is in the optimal capture area. Specifically, in this step, determine whether the total area update count targetnum is greater than or equal to the threshold TargetNum. If not, it means that the total update count is insufficient. All grids are used to represent the optimal capture area. A first mark (which can be 1 or other marks to distinguish it from the second mark, and the first mark below is 1 for example) is assigned to each element of the grid array grid_capture[Width][Height]. If the total area update count targetnum is greater than or equal to the threshold TargetNum, the optimal capture area update count is sufficient. The two-dimensional array data grid_ queue[i][j] is judged. If there is data in the queue that is not the second mark (it can be 0 or other marks, which can be distinguished from the first mark. The second mark below is all exemplified by 0), it means that this area is within the optimal capture area, and the corresponding grid array grid_capture[i][j] is assigned a value of 1. If the queue data is all 0, grid_capture[i][j] is assigned a value of 0, where grid_capture[i][j] indicates whether the area corresponding to the i-th row and j-th column area is the optimal capture area; grid_queue[i][j] indicates the queue corresponding to the i-th row and j-th column area.

[0077] It should be noted that in the above steps of this embodiment, two grids are provided: a grid array, grid_capture, and a grid queue, grid_queue. The grid array represents the optimal capture area; that is, as long as grid_capture[i][j] is equal to 1, the grid at row i, column j is in the optimal capture area. The grid queue, grid_queue, is an intermediate data structure used to generate the grid array. If grid_queue[i][j] represents a non-zero value in the queue of the grid, grid_capture[i][j] is set to 1.

[0078] In step 203, areas with a low update frequency can also be filtered. Specifically, the grid array grid_capture of the best capture area is polled and the update frequency p of each grid is calculated. The larger the update frequency p, the more times the highest target image quality score is updated in this grid. If the update frequency p is less than the preset frequency threshold UpdateThd, the area corresponding to the grid is filtered out. Specifically, the calculation method of the update frequency p includes:

[0079]

[0080] Where capture_targetnum represents the number of 1s in each grid queue, and targetnum represents the total number of 1s and 0s in each grid queue. A higher update frequency p indicates a higher number of highest quality scores updated for this grid. If p is less than the frequency threshold UpdateThd, indicating that the number of best captures in this area is very low, this area is filtered out to prevent it from affecting the best capture area. The specific frequency threshold UpdateThd can be set as needed, for example, 0.1.

[0081] During the capture area update process, this embodiment also provides an optimal capture area update strategy. Specifically: when the camera position is changed or the camera shooting environment changes, if the image difference between the two target detections is less than the preset threshold, it indicates that the image environment has not changed much, and the original optimal capture area will not be cleared. When the target is detected, it will be updated on the original optimal capture area; if the image difference between the two detections is greater than the preset threshold, it indicates that the image environment has changed significantly, and the original optimal capture area will be cleared and the optimal capture area will be regenerated. When the number of target re-detections reaches a predetermined number, the newly generated optimal capture area will be the optimal capture area for the new scene. This strategy can avoid the impact caused by excessive environmental changes. The preset threshold of image difference in this strategy is determined according to needs, for example, it can be set to 50%. It should be noted that the image difference in this embodiment refers to the difference between the entire image when the two captures are taken, rather than the difference between the capture targets. In addition, in addition to using image differences to make judgments, in other embodiments, image similarity can also be used to make opposite judgments. Accordingly, if the image similarity of the two target detections is greater than the preset threshold, it indicates that the image environment has not changed much, and the original best capture area will not be cleared; if the image similarity of the two target detections is less than the preset threshold, it indicates that the image environment has changed significantly, and the original best capture area will be cleared. The definition of image similarity also refers to the similarity of the entire image when the two captures are taken. The operation after judging whether the image environment has changed little or much by image similarity is consistent with the operation after judging the image difference. Clearing the original best capture area in this step specifically assigns 0 to all areas of grid_capture and clears all queues of grid_queue, thus clearing the best capture area that has been formed.

[0082] Reference for the capture area update process including the above-mentioned optimal capture area update strategy Figure 5 As shown, the steps are as follows.

[0083] Step 20: Enter the capture area.

[0084] Step 21: Determine whether the similarity of the entire image between the two target detections is less than a threshold; if not, clear the original area and wait for the capture area input again. If so, continue the capture area update process.

[0085] Step 22: The grid corresponding to the captured area is added to queue 1, and the non-corresponding grid is added to queue 0.

[0086] Step 23: Determine whether the total number of area updates is greater than or equal to a threshold; if so, dequeue all tail data of the grid queue and proceed to step 24; if not, proceed to step 25.

[0087] Step 24: By determining whether each queue in the grid queue contains 1, the grid queue is generated into a grid array, and the process goes to step 26.

[0088] Step 25: Generate a grid array directly and set all areas of the grid array to 1.

[0089] Step 26: Generate the final optimal capture area.

[0090] In order to understand the process of updating the captured area and filtering the low-weight area in step 200 of this embodiment in more detail, this embodiment further illustrates the above process through a specific example. Figure 6 The example diagram of the capture area update process shown in the figure and Figure 7 The example diagram of updating the snapshot area is as follows:

[0091] 1. Object A enters the image from (1,3) and exits from (5,3). Position (3,3) has the highest score. However, since only one capture has been made, which is less than the capture threshold of 2, the entire image is set as the optimal exposure area. Note that in this example, the coordinates (x, y) where x represents the xth column from the left and y represents the xth row from the bottom. Therefore, (1,3) represents the first column from the left and the third row from the bottom. The same applies to the other coordinates in this example.

[0092] 2. Object B enters the screen from (1,4) and exits from (5,4). The score at position (3,4) is the highest, and the capture count 2 is not less than the capture threshold 2. Therefore, (3,3) and (3,4) are set as the best capture areas.

[0093] 3. Object C enters the image from (1,2) and exits from (5,2). The score at position (3,2) is the highest, and the capture count 3 is greater than the capture threshold 2. Therefore, the (3,3) area is removed, and (3,4) and (3,2) are set as the best capture areas.

[0094] 4. Repeat step 3 above for object C 10 times. This means that the frequency of region (3,4) is now 1 / 11, which is less than the frequency threshold of 1 / 10. Region (3,4) is deleted, and region (3,2) is retained. It should be noted that the optimal capture region can actually consist of multiple, disconnected grids; this example only uses a single grid.

[0095] 5. After object A is captured, the image similarity between the captured image and the last captured image of object C is less than the threshold of 50%. At this time, the original best capture area is cleared, so the entire image is set as the best exposure area.

[0096] refer to Figure 8, step 300 of this embodiment (calculating the target average brightness for the target in the optimal capture area, determining whether exposure adjustment is required based on the target average brightness, and performing exposure adjustment if necessary) can be expanded to the following steps.

[0097] Step 301: Output the target area array target_area[num], where each target area array element contains coordinates and width and height (x, y, width, height), and num represents the number of identified targets. In this embodiment, the target area array in this step is output by the target recognition module.

[0098] Step 302: Filter the target. If the target area is not in the optimal capture area, delete the target. The method for determining whether the target area is in the optimal capture area includes: determining whether the overlap degree Overlap between the target area and the optimal capture area is greater than a preset overlap degree threshold OverlapThd. If not, it is determined that the target area is not in the optimal capture area. Specifically, the overlap degree calculation method includes:

[0099]

[0100] Among them, gridnum_cap represents the number of grids in the grid where the target is located that are the best capture area, and gridnum_all represents the total number of grids where the target is located.

[0101] Step 303: The filtered target area array is target_area[filted_num], where filted_num represents the number of filtered targets. The average brightness of the filtered target area is then calculated. In this step, calculating the average brightness of the filtered target area, mean_brightness, specifically includes traversing the filtered target area array and first calculating the average brightness of a single target. The average brightness calculation method for a single target includes obtaining the original image in YUV format, accumulating and averaging the Y values ​​of each pixel in the target area, and then calculating the average brightness of the single target. The average brightness of all the individual targets is then averaged to obtain the final average brightness of the filtered target area.

[0102] Step 304: Determine whether the average brightness of the filtered target area is within the ideal range. If not, adjust the exposure time or gain to optimize the brightness to the ideal range. It should be noted that adjusting the exposure is to adjust the exposure time and exposure gain.

[0103] In summary, the embodiment of the present invention uses a target tracking and target image quality evaluation algorithm to record the target image quality score during the target tracking process, and record the target area with the maximum target image quality score during the target tracking process when the target disappears or times out. Then, the target tracking process within a period of time or a certain number of targets is counted, and the optimal capture area is updated through the area update algorithm. Finally, the detected targets are filtered, the target average brightness of the targets in this area is calculated, and it is determined whether the exposure of this target average brightness needs to be adjusted. If necessary, the exposure adjustment is performed based on the target average brightness in the optimal capture area. Through the improvement of the present invention, the optimal capture area can be focused on to avoid the impact of targets in non-optimal capture areas on exposure quality. In addition, because the present invention can update the optimal capture area according to the prescribed strategy, when the user changes the camera position or the camera shooting environment changes, if the difference between the images of the two target detections is less than the threshold, it indicates that the image environment has not changed much, and the original area will not be cleared. When the target is detected, it will be updated on the original optimal capture area. If the difference between the two detected images is greater than the threshold, it indicates that the image environment has changed significantly. In this case, the original area will be cleared and the optimal capture area will be regenerated. When the number of re-detection targets reaches a predetermined number, the newly generated optimal capture area will be the optimal capture area for the new scene. This strategy can avoid the impact caused by large environmental changes.

[0104] Example 2:

[0105] Based on the implementation method for enhancing the optimal snapshot effect provided in Example 1, this Example 2 provides a specific example in a certain scenario as an example, and reflects the differences and advantages of the present invention by comparing Example 1 of the present invention with different solutions in the prior art.

[0106] like Figure 9 As shown in the figure, for a common scene, the brightness in the middle of the picture is moderate, and the brightness on the left and right sides is darker. Assuming that the target brightness is only affected by the environment and there is no special scene such as backlight, since the environment is bright in the middle and dark on both sides, the target is brighter in the middle of the picture and darker on both sides. In addition, assuming that the object speed is uniform and the image noise is uniform and moderate, the image quality score based on deep learning is mainly affected by the brightness, clarity, noise, etc. of the target. Due to the large number of feature vectors, it is difficult to use a complete formula to express the quality scoring algorithm. If you want to use a simple formula analogy, you can use the following formula to express the scoring calculation:

[0107] Q = W0*|Luminance+L0|+W1*Definition+w3*Other. W0 represents the luminance weight, Luminance represents the luminance feature, which is obtained by calculating the average luminance of the target area, and L0 represents the luminance feature offset. W1 represents the clarity weight, Definition represents the clarity feature, w3 represents the weight of other features, and Other represents other features.

[0108] This formula shows that the more moderate the brightness, the greater the image quality score. Too high or too low image brightness will result in a decrease in image quality score. Under the same brightness, the greater the clarity Definition, the greater the score. Definition is often negatively correlated with the exposure multiple Expgain. The greater the exposure multiple, the greater the noise, the more blurred the moving object tends to be, and the smaller the clarity Definition. Assuming that the target optimal brightness is 90, being higher than 90 or lower than 90 will result in a decrease in the score. If the brightness is increased by increasing the exposure multiple, the score will increase in brightness and decrease in clarity. The influence weight of brightness is often greater than the influence weight of clarity, so the overall score will increase. In order to be more intuitive, a possible image quality scoring formula is used (it should be noted that the FaceQnet algorithm is too complicated and is not the innovation of this embodiment. Therefore, this embodiment does not explain the FaceQnet algorithm in detail. The following formula is a simplified version of the image quality scoring formula used as an example in this embodiment):

[0109] Q = 7-1 / 10*|Luminance-90|-ExpGain, where W0 = -1 / 10, W1 = -1, and W3*other = 7. This formula is used for the following three processing methods.

[0110] There are three ways to handle partial exposure of this scene:

[0111] Solution 1: No local exposure is performed, and the target brightness is affected by the environment. Assuming that targets A and B have the same brightness in the same location, the following scoring results will be obtained.

[0112] In the first frame, Target A has a brightness of 30, which is relatively dim, so the image quality score is 1. In the second frame, Target A has a brightness of 40, and an image quality score of 2. In the third frame, Target B begins to enter, and the frame now contains both Target A and Target B. Target A has a brightness of 60, and an image quality score of 4, while Target B has a brightness of 30, and an image quality score of 1. In the fourth frame, Target A has a brightness of 50, and an image quality score of 3, while Target B has a brightness of 40, and an image quality score of 2. In the fifth frame, Target A has a brightness of 40, and an image quality score of 2, while Target B has a brightness of 60, and an image quality score of 4. In the sixth frame, Target A exits the frame, while Target B remains in the frame. At this time, Target B has a brightness of 50, and an image quality score of 3. In the final frame, Target B has a brightness of 40, and an image quality score of 2.

[0113] From the first frame to the seventh frame, the maximum score of target A is 4, and the maximum score of target B is 4.

[0114] Option 2: The existing patent performs partial exposure on all targets.

[0115] In the first frame, the original brightness of target A is 30. The target detection system detects the target and applies exposure compensation, increasing the target brightness to 90 by adjusting the exposure factor. While the target brightness is now appropriate, the quality score decreases in clarity as the exposure factor increases. Therefore, the exposure factor for the first frame is 3x, resulting in a score of Q = 7 - 3 = 4. The exposure factor for the second frame is increased to 2.25x, resulting in a score of 7 - 2.25 = 4.75. In the third frame, due to the arrival of object B, the existing patent takes the average brightness of targets A and B, which is 45. Using this average as a guide, the exposure is adjusted to 90, meaning the exposure factor is increased to 2x. At this point, target A's brightness is 120, resulting in a score of Q = 7 - |90 - 120| / 10 - 2 = 2. Target B's brightness is 60, resulting in a score of Q = 7 - |60 - 90| / 10 - 2 = 2. Similarly, in the fourth frame, the brightness of target A is 100, and the score is Q = 7 - |90 - 100| / 10 - 2 = 4. The brightness of target B is 80, and the score is Q = 7 - |90 - 80| / 10 - 2 = 4. In the fifth frame, the exposure factor is increased to 1.8 times, the brightness of target A is 72, and the score is Q = 7 - |72 - 90| / 10 - 1.8 = 3.4. The brightness of target B is 108, and the score is Q = 7 - |108 - 90| / 10 - 1.8 = 3.4. In the sixth frame, the exposure factor is increased to 2.25 times, the brightness of target B is 90, and the score is Q = 7 - 1.8 = 5.2. In the seventh frame, the brightness of target A is 90, and the score is Q = 7 - 2.25 = 4.75.

[0116] During this process, the maximum image quality score of target A was 4.75, and the maximum image quality score of target B was 5.2.

[0117] Solution 3: In the embodiment of the present invention, local exposure is performed on the target in the optimal capture area.

[0118] In the first frame, object A enters the frame. Since it's not in the optimal capture area, no partial exposure is performed. Therefore, the brightness is 30, and the score is Q = 7 - |30 - 90| / 10 = 1. In the second frame, object A's brightness is 40, and the score is Q = 7 - |40 - 90| / 10 = 2. In the third frame, object B enters the frame, and object A enters the optimal capture area. Therefore, a partial exposure is performed on object A, and the exposure factor is increased to 1.5. Object A's score is Q = 7 - 1.5 = 5.5, and object B's score is Q = 7 - |45 - 90| / 10 - 1.5 = 1. In the fourth frame, object A exits the optimal capture area, but object B has not yet entered it. Therefore, the exposure factor remains the same, 1.5. Object A's score is now Q = 7 - |75 - 90| / 10 - 1.5 = 4, and object B's score is Q = 7 - |60 - 90| / 10 - 1.5 = 2.5. In the fifth frame, object B enters the optimal capture area. A partial exposure of object B is performed at an exposure factor of 1.5x. At this point, object A's score Q = 7 - |60 - 90| / 10 - 1.5 = 2.5, and object B's score Q = 7 - 1.5 = 5.5. In the sixth frame, object B is no longer in the optimal capture area. The exposure factor of 1.5x is used again. Object B's brightness is 75, and its quality score is Q = 7 - |75 - 90| / 10 - 1.5 = 4. In the seventh frame, object B's brightness is 60, and its quality score is Q = 7 - |60 - 90| / 10 - 1.5 = 2.5.

[0119] In this scenario, the maximum quality score of objective A is 5.5, and the maximum quality score of objective B is 5.5, both of which are higher than those of scenarios 1 and 2.

[0120] From the above comparison, it can be seen that the method provided by the embodiment of the present invention has more advantages than the existing technology and has better capture quality.

[0121] Example 3:

[0122] Based on the method for implementing the enhanced optimal snapshot effect provided in the above embodiment 1, the present invention also provides an implementation device for implementing the enhanced optimal snapshot effect of the above method and system, such as Figure 10 , is a schematic diagram of the device architecture of an embodiment of the present invention. The device for enhancing the optimal snapshot effect of this embodiment includes one or more processors 21 and a memory 22. Figure 10 A processor 21 is taken as an example.

[0123] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 10 The bus connection is taken as an example.

[0124] Memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the method for enhancing the optimal snapshot effect in Example 1. Processor 21 executes the non-volatile software programs, instructions, and modules stored in memory 22 to execute various functional applications and data processing of the device for enhancing the optimal snapshot effect, thereby implementing the method for enhancing the optimal snapshot effect in Example 1.

[0125] The memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor 21, and such remote memory may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the method for implementing the enhanced optimal snapshot effect in the above embodiment 1 is executed. For example, the above described method is executed. Figure 1-Figure 5 、 Figure 8 The steps shown.

[0127] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a disk or an optical disk, etc.

[0128] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention. Any matters not described in detail in this specification constitute prior art known to those skilled in the art.

Claims

1. A method for enhancing the optimal snapshot effect, characterized in that: include: During the target tracking process, the target image quality score of the target is recorded. When the target disappears or times out, the target area with the maximum target image quality score during the target tracking process is recorded. Statistics the target tracking process within a period of time or a certain number of targets, and update the best capture area; The statistics of the target tracking process within a period of time or a certain number of targets, and the updating of the best capture area specifically include: dividing the images in the target tracking process within a period of time or a certain number of targets into preset grids, using the two-dimensional array data grid_queue to save the capture order of different target areas, each two-dimensional array element maintains a queue, the maximum length of the queue is the threshold TargetNum, and the threshold TargetNum represents the number of captures required to form the best capture area; each time the best capture area needs to be updated, the queue corresponding to the grid of the target area with the highest target image quality score is put into the first mark, and the queue of the non-corresponding grid is put into the second mark, and it is judged whether the total number of area updates targetnum is greater than the threshold TargetNum. If so, all the grid tail data are dequeued, otherwise they are only put into the queue but not dequeued; the grid array grid_capture is used to represent the best capture area, and the value of each grid array element represents whether this target area is in the best capture area; the use of the grid array grid_capture to represent the best capture area, and the value of each grid array element represents Whether this target area is in the best capture area specifically includes: judging whether the total area update number targetnum is greater than or equal to the threshold TargetNum. If it is not greater than or equal to, it means that the total update number is insufficient, all grids are used to represent the best capture area, and each element of the grid array grid_capture is assigned a first mark; if the total area update number targetnum is greater than or equal to the threshold TargetNum, the best capture area update number is sufficient, and the two-dimensional array data grid_queue[i][j] representing the grid queue is judged. If there is data in the queue that is not the second mark, it means that this area is in the best capture area, and the corresponding grid array grid_capture[i][j] is assigned a first mark. If the queue data are all the second mark, grid_capture[i][j] is assigned a second mark, wherein grid_capture[i][j] indicates whether the area corresponding to the area of ​​the i-th row and j-th column is the best capture area; grid_queue[i][j] represents the queue corresponding to the area of ​​the i-th row and j-th column; Calculate the average brightness of the target in the optimal capture area, and determine whether exposure adjustment is needed based on the target average brightness. If necessary, perform exposure adjustment.

2. The method for enhancing the optimal snapshot effect according to claim 1, characterized in that: The acquisition of the target image quality score specifically includes: Use the target recognition algorithm to identify the target in the image and output the target area array after recognition. The target area array represents the location information of all detected targets. Each target area element in the target area array contains coordinates and width and height; The original image is segmented by the coordinates and width and height of the target area array to obtain the target image array to represent the target images of all detected targets; A target image quality score algorithm is used to perform target image quality scores on all target image arrays to obtain a target quality score array to represent the target image quality scores of all detected targets.

3. The method for enhancing the optimal snapshot effect according to claim 2, characterized in that: The target image quality score of the target is recorded during the target tracking process, and the target area with the maximum target image quality score is recorded during the target tracking process when the target disappears or times out. Specifically, the target image quality score of the target is recorded during the target tracking process. The target is tracked using a target tracking algorithm. The tracking process repeats the acquisition of the target area array, the target image array, and the target quality score array, while simultaneously capturing the target. If the target is captured for the first time, the target area and target image quality score of this capture are recorded; if the target is not captured for the first time, it is determined whether the target image quality score of the target in this tracking process is greater than the target image quality score of the previous time. If so, the target area and target image quality score of the previous time are replaced with the target area and target image quality score of the current time; When the target tracking algorithm determines that the target disappears or the residence time exceeds the preset threshold, the target area with the highest target image quality score during the tracking process is output for subsequent optimal capture area update.

4. The method for enhancing the optimal snapshot effect according to claim 1, characterized in that: The method for determining whether a grid is the target area with the highest target image quality score specifically includes: Determine the grids where the coordinates of the four corners of the target area with the highest target image quality score are located. If the four corners are in different grids, four grids will be formed, and the rectangular grid area surrounded by these four grids will be used as the corresponding grid area of ​​the target area with the highest target image quality score. If the four corners are in only two grids or only one grid, the two grids and the area between them or the single grid will be used as the corresponding grid area of ​​the target area with the highest target image quality score.

5. The method for enhancing the optimal snapshot effect according to claim 1, characterized in that: It also includes filtering areas with low update frequencies. Specifically, the grid array grid_capture of the best capture area is polled, and the update frequency p of each grid is calculated. The larger the update frequency p, the more times the highest target image quality score is updated in this grid. If the update frequency p is less than the preset frequency threshold UpdateThd, the area corresponding to the grid is filtered out.

6. The method for enhancing the optimal snapshot effect according to claim 1, characterized in that: Calculating the target average brightness of the target in the optimal capture area, determining whether exposure adjustment is required based on the target average brightness, and performing exposure adjustment if necessary specifically includes: Output the target area array target_area[num], each target area array element contains coordinates and width and height (x, y, width, height), num represents the number of recognized targets; Filter the target, and if the target area is not in the optimal capture area, delete the target; the method of determining whether the target area is in the optimal capture area includes: determining whether the overlap between the target area and the optimal capture area is greater than a preset overlap threshold; if not, determining that the target area is not in the optimal capture area; The array of the filtered target area is target_area[filted_num], where filted_num represents the number of filtered targets. The average brightness of the filtered target area is then calculated. Determine whether the average brightness of the filtered target area is within the ideal range. If not, optimize the brightness to the ideal range by adjusting the exposure time or gain.

7. The method for enhancing the optimal snapshot effect according to claim 6, characterized in that: The calculation of the average brightness of the filtered target area specifically includes: Traverse the filtered target area array and first calculate the average brightness of a single target. The average brightness calculation method of a single target includes: obtaining the original image in YUV format, accumulating and averaging the Y value of each pixel in the target area to obtain the average brightness of the single target; then perform brightness averaging calculation on the average brightness of all single targets to obtain the final average brightness of the filtered target area.

8. The method for enhancing the optimal snapshot effect according to any one of claims 1 to 7, characterized in that: When the camera position is changed or the camera shooting environment changes: If the difference between the two target detection images is less than the preset threshold, it indicates that the image environment has not changed much, and the original optimal capture area will not be cleared. When the target is detected, it will be updated on the original optimal capture area. The image difference refers to the difference between the two captures. If the difference between the images detected before and after is greater than the preset threshold, it indicates that the image environment has changed significantly. In this case, the original best capture area will be cleared and a new best capture area will be generated. When the number of re-detection targets reaches the set number, the newly generated best capture area will be the best capture area for the new scene.

9. A device for enhancing optimal snapshot effect, characterized by: It includes at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the method for implementing the enhanced optimal snapshot effect as described in any one of claims 1-8.

Images