Video frame image processing method and device, electronic equipment and storage medium

By determining the crop boundary according to resolution limits in the video frame image and cropping multiple times, sub-images are generated for target recognition, the problem of image quality reduction and target damage caused by high-resolution image processing in the prior art is solved, and video analysis is improved Applicability and identification accuracy of the equipment.

CN119027634APending Publication Date: 2024-11-26JINAN YUSHI INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310596699.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When processing high-resolution video frame images, pixel sampling or image segmentation methods can easily lead to a decrease in image quality, resulting in unrecognition of targets or reduced recognition accuracy, and the targets may be damaged or lost due to cropping.

Method used

By determining the crop boundary for first and second cropping according to the resolution limits of the video frame image and the video analysis device, multiple sub-images are generated, and target recognition is performed while ensuring that the crop boundary is within the sub-image.

Benefits of technology

While ensuring the quality of video frame images, the applicability of video analysis equipment to high-resolution images is improved, and the problem of missing target detection and reduced recognition accuracy is avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027634A_ABST
    Figure CN119027634A_ABST
Patent Text Reader

Abstract

The invention discloses a video frame image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: determining a clipping boundary of first clipping according to the resolution of a video frame image and the resolution limitation of video analysis equipment; according to the cutting boundary of the first cutting, performing first cutting on the to-be-processed video frame image to obtain at least two first cutting sub-images; according to a cutting boundary when the video frame image to be processed is cut for the first time, the video frame image to be processed is cut for the second time, at least two secondary cutting sub-images are obtained, and the cutting boundary when the video frame image is cut for the first time is located in the secondary cutting sub-images; and performing target identification according to each first-time cutting sub-image and each second-time cutting sub-image. By using the technical scheme provided by the invention, the applicability of video analysis equipment to high-resolution images can be improved while the quality of video frame images is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for processing video frame images. Background Art

[0002] With the continuous development of the security field towards intelligence and high-resolution video, the resolution of video frame images captured by front-end shooting equipment can often reach several million or even more than 10MP, while the image size that can be processed by back-end video analysis equipment is often smaller than the size of the video frame image captured by the front-end shooting equipment, generally only 2MP.

[0003] In the prior art, in order to improve the adaptability of the back-end video analysis equipment to high-resolution images, high-resolution images are generally scaled by pixel sampling, or the high-resolution images are segmented into sizes available to the video analysis equipment by image segmentation. However, the pixel sampling method will lead to a decrease in the quality of the video frame image, resulting in the problem of unrecognized targets or reduced recognition accuracy, while the image segmentation method is likely to cause the target in the video frame image to be damaged or lost due to cropping. Summary of the invention

[0004] The present invention provides a method, device, electronic device and storage medium for processing video frame images, so as to improve the applicability of video analysis equipment to high-resolution images while ensuring the quality of video frame images.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing a video frame image, the method comprising:

[0006] Determine a cropping boundary for the first cropping according to the resolution of the video frame image and the resolution limit of the video analysis device;

[0007] Performing a first cropping on the video frame image to be processed according to the cropping boundary of the first cropping to obtain at least two first cropped sub-images;

[0008] Performing secondary cropping on the video frame image to be processed according to the cropping boundary during the first cropping of the video frame image to be processed, to obtain at least two secondary cropped sub-images, wherein the cropping boundary during the first cropping is located within the secondary cropped sub-images;

[0009] Target recognition is performed based on each first-cropped sub-image and each second-cropped sub-image.

[0010] In a second aspect, an embodiment of the present invention further provides a video frame image processing device, the device comprising:

[0011] A cropping boundary determination module, used to determine a cropping boundary for the first cropping according to the resolution of the video frame image and the resolution limit of the video analysis device;

[0012] A first cropping module, used for performing a first cropping on the video frame image to be processed according to a first cropping boundary, to obtain at least two first cropped sub-images;

[0013] A secondary cropping module, used for performing secondary cropping on the video frame image to be processed according to the cropping boundary during the first cropping of the video frame image to be processed, so as to obtain at least two secondary cropped sub-images, wherein the cropping boundary during the first cropping is located within the secondary cropped sub-images;

[0014] The target recognition module is used to perform target recognition based on each first cropped sub-image and each second cropped sub-image.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for processing a video frame image as described in any one of the embodiments of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute a method for processing a video frame image as described in any one of the embodiments of the present invention.

[0017] The technical solution of the embodiment of the present invention determines the cropping boundary of the first cropping according to the resolution of the video frame image and the resolution limit of the video analysis device, performs the first cropping on the processed video frame image according to the cropping boundary of the first cropping, obtains at least two first cropped sub-images, and then performs the second cropping on the processed video frame image according to the cropping boundary when the processed video frame image is first cropped, obtains at least two second cropped sub-images, the cropping boundary is located within the second cropped sub-image, and performs target recognition according to each first cropped sub-image and each second cropped sub-image. The problem in the prior art that the method of processing high-resolution images through pixel sampling will cause the quality of the video frame image to be reduced, thereby causing the target to be not recognized or the recognition accuracy to be reduced, and the problem that the method of processing high-resolution images through image segmentation is easy to cause the target to be damaged or lost due to cropping is solved, and the applicability of the video analysis device to high-resolution images is improved while ensuring the quality of the video frame image.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flowchart of a method for processing a video frame image provided by Embodiment 1 of the present invention;

[0021] Figure 2 is a schematic diagram of first cropping of a video frame image provided by the first embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of secondary cropping of a video frame image provided by the first embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of another method of secondary cropping of a video frame image provided by the first embodiment of the present invention;

[0024] Figure 5 is a flowchart of a method for processing a video frame image provided by Embodiment 2 of the present invention;

[0025] Figure 6 is a structural schematic diagram of a video frame image processing device provided by Embodiment 3 of the present invention;

[0026] Figure 7 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 A flowchart of a method for processing a video frame image is provided for the first embodiment of the present invention. This embodiment can be applied to the situation where high-resolution video frame images are processed in order to improve the adaptability of video analysis equipment to high-resolution images. The method can be executed by a video frame image processing device. The video frame image processing device can be implemented in the form of hardware and / or software. The video frame image processing device can be configured in an electronic device and used in conjunction with the video analysis equipment.

[0031] like Figure 1 As shown, the method includes:

[0032] S110: Determine a cropping boundary for the first cropping according to the resolution of the video frame image and the resolution limit of the video analysis device.

[0033] Among them, the resolution limit of the video analysis device depends on the resolution size of the image that can be processed by the video analysis device. For example, the resolution limit may be 1980*1020, and the corresponding image size is about 2MP. However, for some high-resolution video frame images, the image size corresponding to the resolution thereof may even reach 8MP or even 10MP or more. Therefore, this embodiment crops the video frame images whose resolution is greater than the resolution limit so that the cropped images meet the resolution limit of the video analysis device.

[0034] The cropping boundary of the first cropping refers to the boundary area formed by the cropping line when the video frame image is first cropped. In this embodiment, the function of determining the cropping boundary of the first cropping is to facilitate the subsequent secondary cropping of the video frame image according to the cropping boundary, thereby avoiding target leakage.

[0035] In this embodiment, the cropping boundary of the first cropping is determined according to the resolution of the video frame image and the resolution limit of the video analysis device. The sizes of the first cropped sub-images after the first cropping can be the same or different. The image size of the cropped sub-image after the first cropping must meet the resolution limit of the video analysis device.

[0036] S120 . Perform a first cropping on the video frame image to be processed according to the first cropping boundary to obtain at least two first cropped sub-images.

[0037] The sizes of the first-cropped sub-images after the first cropping are the same and / or different.

[0038] Furthermore, if the sizes of the sub-images cropped for the first time after the first cropping are different, the length size of the sub-images cropped for the first time after the first cropping meets the length resolution limit of the video analysis device, and the height size of the sub-images cropped for the first time after the first cropping meets the height resolution limit of the video analysis device.

[0039] The size of the first cropped sub-image can be flexibly set according to the needs of target recognition, but the size of the first cropped sub-image must meet the resolution limit of the video analysis device so that the video analysis device can process the first cropped sub-image.

[0040] In a specific example where the sizes of the first cropped sub-images are different, the location area where the target appears more often can be determined based on the target locations in multiple historical video frame images. In this case, the first cropped sub-image can be divided into the first cropped sub-image containing the location area, and other first cropped sub-images obtained by dividing other areas outside the location area in the video frame image. The sizes of the first cropped sub-image containing the location area and other first cropped sub-images can be different.

[0041] Furthermore, if the sizes of the sub-images cropped for the first time are the same after the first cropping, S110 may include: determining the number of equal divisions of the video frame image according to the ratio of the resolution of the video frame image to the resolution limit of the video analysis device; and determining the cropping boundary of the first cropping according to the size of the video frame image and the number of equal divisions.

[0042] The number of equal divisions indicates the number of each cropped sub-image after the first cropping, that is, the number of equal divisions of the video frame image.

[0043] In an example, taking the image size of a video frame image as 8MP and the image size corresponding to the resolution limit of a video analysis device as 2MP, it can be determined that the number of equal divisions is 4, the length of the first cropped sub-image is half the length of the video frame image, and the height of the first cropped sub-image is half the height of the video frame image. Figure 2 A schematic diagram of the first cropping of a video frame image is provided, such as Figure 2 As shown, by cropping the video frame image along the midline in the length direction and the midline in the height direction, four first cropped sub-images of the same size can be obtained.

[0044] In another example, taking the image size of a video frame image as 10MP and the image size corresponding to the resolution limit of the video analysis device as 2MP, 10MP / 2MP=5, 4 (two equal divisions along the length direction and the height direction)<5<9 (three equal divisions along the length direction and the height direction), so the number of equal divisions corresponding to the 10MP video frame image is 9. That is, when the video frame image is divided into two equal divisions along the length direction and the height direction, the number of equal divisions is 4, and the image size of the four cropped sub-images obtained is 2.5MP, which does not meet the resolution limit. Therefore, the video frame image is divided into three equal divisions along the length direction and the height direction, that is, the number of equal divisions is 9, to ensure that the nine cropped sub-images obtained meet the resolution limit. At this time, the length of the first cropped sub-image is one-third of the length of the video frame image, and the height of the first cropped sub-image is one-third of the height of the video frame image.

[0045] It should be noted that, in this embodiment, the same number of equal divisions of the video frame image along the length direction and along the height direction are used as an example for explanation. In this case, the image aspect ratio of the cropped sub-image after cropping is the same as that of the original video frame image, which is convenient for the video analysis device to process. The number of equal divisions of the video frame image along the length direction and the number of equal divisions along the height direction may also be different, for example, the video frame image is divided into three equal parts along the length direction and into two equal parts along the height direction, or the video frame image is divided into two equal parts along the length direction and into three equal parts along the height direction, to obtain six cropped sub-images, and this embodiment does not limit this.

[0046] S130 , performing secondary cropping on the video frame image to be processed according to the cropping boundary when the video frame image to be processed is cropped for the first time, to obtain at least two secondary cropped sub-images.

[0047] The cropping boundary during the first cropping is located within the second cropping sub-image.

[0048] by Figure 2 For example, Figure 2The video frame image includes target 1 and target 2. After cropping the video frame image, target 2 located at the lower right side of the video frame image can still be completely identified. However, since target 1 located on the left side of the video frame image just intersects with the midline of the video frame image along the height direction, if target recognition is performed only on the first cropped sub-image, only target 2 can be identified. Therefore, in order to avoid the situation where the target near the cropping boundary cannot be detected due to being cut by the cropping boundary when cropping the video frame image, this embodiment performs a second cropping on the video frame image, and the cropping boundaries during the first cropping must all be located within the image area composed of the second cropped sub-image to avoid missed detection due to cropping of the target.

[0049] Furthermore, the sizes of the secondary cropped sub-images after the secondary cropping are the same and / or different; the sizes of the secondary cropped sub-images meet the resolution limitation of the video analysis device; the sizes of the secondary cropped sub-images are the same and / or different from the sizes of the first cropped sub-images, and the distance from the cropping boundary to the edge of the secondary cropped sub-image containing the cropping boundary and parallel to the cropping boundary is greater than or equal to the standard target size.

[0050] The size of the secondary cropped sub-image may be different from the size of the first cropped sub-image, and the sizes of the secondary cropped sub-images may be the same or different. This embodiment does not limit the size of the secondary cropped sub-image, but the size of the secondary cropped sub-image must at least be able to ensure that when there is a target near the cropping boundary, the target can be completely detected. Therefore, the distance from the cropping boundary to the edge parallel to the cropping boundary in the secondary cropped sub-image containing the cropping boundary must be greater than or equal to the standard target size. The standard target size can be determined based on an empirical value or based on the historical target size in the historical target recognition result. This embodiment does not limit this.

[0051] Figure 3 A schematic diagram of secondary cropping of video frame images is provided. Figure 3 For example, the cropped sub-images are numbered at the lower left corner of the cropped sub-images. (1), (2), (3) and (4) are the first cropped sub-images. Figure 3 The second cropping obtains two second cropped sub-images (5) and (6). The distance from the center line in the length direction of the video frame image to the edge of the second cropped sub-image (5) in the length direction must be greater than or equal to the standard target size, and the distance from the center line in the height direction to the edge of the second cropped sub-image (6) in the height direction must be greater than or equal to the standard target size.

[0052] At the same time, it should be noted that the size of the secondary cropped sub-image should also comply with the resolution limit of the video analysis device so that the video analysis device can detect and identify the secondary cropped sub-image.

[0053] Further, S130 may include: determining a cropping line segment and a cropping intersection based on a cropping boundary when the video frame image to be processed is cropped for the first time; performing a secondary cropping on the video frame image to be processed based on the cropping line segment and the cropping intersection to obtain at least two secondary cropped sub-images; wherein the cropping line segment is a midline of the secondary cropped sub-image containing the cropping line segment, and the cropping intersection is a midpoint of the secondary cropped sub-image containing the cropping intersection.

[0054] Figure 4 Another schematic diagram of secondary cropping of video frame images is provided, such as Figure 4 As shown, when the video frame image is cropped for the first time, the midline along the length direction and the midline along the height direction are cropped. The midline along the length direction and the midline along the height direction in the video frame image are the cropping boundaries for the first cropping. The midline along the length direction and the midline along the height direction intersect to form four cropping line segments and a cropping intersection. The cropped sub-image is numbered at the lower left corner of the cropped sub-image. Figure 4 In (1), (2), (3) and (4) are the first cropped sub-images. To ensure that all cropping boundaries are within the image area composed of all the second cropped sub-images, Figure 4 The secondary cropping results in five secondary cropped sub-images, namely (5), (6), (7), (8) and (9). Figure 4 In the example, the size of the second cropped sub-image is the same as the size of the first cropped sub-image, and the size of each second cropped sub-image is the same. This setting facilitates the video analysis device to identify and detect the image. Figure 4 As shown, the cropping line segment on the left is the midline of the secondary cropping sub-image (5) in the length direction, the cropping line segment on the lower side is the midline of the secondary cropping sub-image (6) in the height direction, the cropping line segment on the right side is the midline of the secondary cropping sub-image (7) in the length direction, the cropping line segment on the upper side is the midline of the secondary cropping sub-image (8) in the height direction, and the cropping intersection is the midpoint of the secondary cropping sub-image (9).

[0055] The technical solution of this embodiment performs secondary cropping based on the intersection of the cropping line segments and line segments contained in the cropping boundary. Take the case where the number of times the video frame image is cropped along the length direction and along the height direction is the same during the first cropping as an example, and x represents the number of times the video frame image is cropped along the length direction, then a total of (x+1)×2x cropping line segments are formed, and x 2Therefore, in order to make the intersection points of the cropping line segments all within the secondary cropping sub-image, the number of secondary cropping sub-images required is (x+1)×2x+x 2 .

[0056] S140: Perform target recognition according to each first-cropped sub-image and each second-cropped sub-image.

[0057] In this embodiment, each first cropped sub-image and each second cropped sub-image that meet the resolution limit of the video analysis device are subjected to target detection and target recognition, thereby improving the adaptability of the video analysis device to high-resolution images. At the same time, since the video frame image is cropped twice, after the first cropping, the second cropping is performed according to the cropping boundary of the first cropping, so that the cropping boundary is located within the image area composed of all the second cropped sub-images, thereby avoiding the situation where the target near the cropping boundary is missed due to being cut by the cropping boundary, thereby ensuring the target detection rate and the accuracy of target detection.

[0058] The technical solution of the embodiment of the present invention determines the cropping boundary of the first cropping according to the resolution of the video frame image and the resolution limit of the video analysis device, performs the first cropping on the processed video frame image according to the cropping boundary of the first cropping, obtains at least two first cropped sub-images, and then performs the second cropping on the processed video frame image according to the cropping boundary when the processed video frame image is first cropped, obtains at least two second cropped sub-images, the cropping boundary is located within the at least two second cropped sub-images, and performs target recognition according to each first cropped sub-image and each second cropped sub-image. The problem in the prior art that the method of processing high-resolution images through pixel sampling will cause the quality of the video frame image to be reduced, thereby causing the target to be not recognized or the recognition accuracy to be reduced, and the problem that the method of processing high-resolution images through image segmentation is easy to cause the target to be damaged or lost due to cropping is solved, and the applicability of the video analysis device to high-resolution images is improved while ensuring the quality of the video frame image.

[0059] Embodiment 2

[0060] Figure 5 A flowchart of a method for processing a video frame image provided in the second embodiment of the present invention. Based on the above embodiments, the embodiment of the present invention further specifies the process of determining the number of equal divisions, the secondary cropping process, and performing target detection and target recognition based on each first cropped sub-image and each second cropped sub-image.

[0061] like Figure 5 As shown, the method includes:

[0062] S210: Determine the number of equal divisions of the video frame image according to a ratio of the resolution of the video frame image to a resolution limit of a video analysis device.

[0063] This embodiment is described by taking the case where the sizes of the sub-images cropped for the first time are the same as an example. The specific process of determining the number of equal divisions has been described in the above embodiment, and will not be repeated in this embodiment.

[0064] S220: Determine a cropping boundary for the first cropping according to the size of the video frame image and the number of equal divisions.

[0065] The manner of determining the cropping boundary for the first cropping according to the size of the video frame image and the number of equal divisions has been described in the above embodiment and will not be repeated in this embodiment.

[0066] S230 , performing a first cropping on the video frame image to be processed according to the first cropping boundary to obtain at least two first cropped sub-images.

[0067] The process of performing the first cropping on the video frame image to be processed has been described in the above embodiment, and will not be repeated in this embodiment.

[0068] S240 , performing secondary cropping on the video frame image to be processed according to the cropping boundary when the video frame image to be processed is cropped for the first time, to obtain at least two secondary cropped sub-images.

[0069] The specific manner of performing secondary cropping on the video frame image to be processed so that the cropping boundary during the first cropping is located within at least two secondary cropped sub-images has been described in the above embodiment, and will not be repeated in this embodiment.

[0070] S250 , performing target detection according to each first cropped sub-image and each second cropped sub-image to obtain a first target detection result and a second target detection result.

[0071] Since each first cropped sub-image after the first cropping and each second cropped sub-image after the second cropping meet the resolution limit of the video analysis device, the video analysis device can perform target detection on them. Therefore, target detection is performed on each first cropped sub-image to obtain a first target detection result, and target detection is performed on each second cropped sub-image to obtain a second target detection result.

[0072] S260: Compare the first target detection result and the second target detection result to determine the comparison result.

[0073] In this embodiment, the reason for comparing and removing duplicates of the first target detection result and the second target detection result is to avoid wasting target recognition performance when the target detected in the first cropped sub-image and the target detected in the second cropped sub-image are the same target.

[0074] Further, S260 may include: for the first target detection result and the second target detection result, respectively determining the image number of the cropped sub-image, the coordinates in the cropped sub-image, and the coordinate conversion relationship between the cropped sub-image corresponding to the cropped sub-image and the video frame image to be processed; performing coordinate conversion on the first target detection result according to the coordinates of the first target detection result in the first cropped sub-image and the coordinate conversion relationship between the cropped sub-image corresponding to the first cropped sub-image and the video frame image to be processed, to obtain the coordinates of the first target detection result in the video frame image to be processed; performing coordinate conversion on the second target detection result according to the coordinates of the second target detection result in the second cropped sub-image and the coordinate conversion relationship between the cropped sub-image corresponding to the second cropped sub-image and the video frame image to be processed, to obtain the coordinates of the second target detection result in the video frame image to be processed; comparing the coordinates of the first target detection result in the video frame image to be processed with the coordinates of the secondary target detection result in the video frame image to be processed, to obtain a comparison result.

[0075] In this embodiment, the image numbers of each first cropped sub-image and each second cropped sub-image are respectively determined in advance, as well as the coordinate conversion relationship between the cropped sub-image and the to-be-processed video frame image.

[0076] by Figure 4 Taking the first cropped sub-images and the second cropped sub-images in as an example, the first cropped sub-images can be numbered first, and then the second cropped sub-images can be numbered. The numbers of the first cropped sub-images and the second cropped sub-images are continuous. The first cropped sub-images and the second cropped sub-images can also be numbered according to their positions in the video frame image. The first cropped sub-images and the second cropped sub-images can also be numbered separately. This embodiment does not limit the numbering method. Taking the first numbering method as an example, any first cropped sub-image can be used as the starting point for numbering, and numbered clockwise or counterclockwise, and then any second cropped sub-image can be used as the starting point for numbering, and numbered clockwise or counterclockwise, and the last digit of the first cropped sub-image number and the first digit of the second cropped sub-image number are continuous numbers. In this embodiment, the image number is determined for each first cropped sub-image and each second cropped sub-image, which facilitates the subsequent deduplication of the target detection results of each cropped sub-image.

[0077] Specifically, determining the coordinate transformation relationship between the cropped sub-image and the video frame image to be processed may include: determining a calibration point of the cropped sub-image, and determining the coordinate transformation relationship between the cropped sub-image and the video frame image to be processed according to the coordinates of the calibration point in the cropped sub-image and the coordinates of the calibration point in the video frame image to be processed.

[0078] The calibration point may be any point in the cropped sub-image, for example, the calibration point may be the lower left corner vertex of the cropped sub-image, or the midpoint of the cropped sub-image. The present embodiment does not limit the selection method of the calibration point. According to the position coordinates of the calibration point in the cropped sub-image and the position coordinates of the calibration point in the video frame image to be processed, the coordinate transformation relationship corresponding to the cropped sub-image may be determined.

[0079] In this embodiment, the coordinates of the first target detection result in the cropped sub-image are converted to the coordinates in the video frame image to be processed, and the coordinates of the second target detection result in the cropped sub-image are converted to the coordinates in the video frame image to be processed, so that the coordinates of the first target detection result and the second target detection result are in the same coordinate system, which is convenient for comparison.

[0080] S270: If it is determined according to the comparison result that there are duplicate targets, deduplication processing is performed on the first target detection result and the second target detection result corresponding to the duplicate target.

[0081] When there are repeated targets, that is, a target detected in the first cropped sub-image and a target detected in the second cropped sub-image are the same target, the repeated target detection results are removed and one target detection result is retained for subsequent target recognition or analysis processing to avoid performance waste.

[0082] Further, S270 may include: if it is determined that the coordinates of the first target detection result to be processed in the video frame image to be processed are the same as the coordinates of the second target detection result to be processed in the video frame image to be processed, then it is determined that there is a duplicate target; according to the resolution of the first target detection result to be processed, the image number of the cropped sub-image where the first target detection result to be processed is located, the resolution of the secondary target detection result to be processed, and the image number of the cropped sub-image where the secondary target detection result to be processed is located, the first target detection result to be processed and the second target detection result to be processed are deduplicated.

[0083] In this embodiment, since the coordinates of the first target detection results and the secondary target detection results are transformed, if the coordinates of the first target detection results to be processed are the same as the coordinates of the secondary target detection results to be processed, the first target detection results to be processed and the secondary target detection results to be processed correspond to the same target.

[0084] Furthermore, when it is determined that there are duplicate targets, deduplication is performed based on the resolution size and the image number of the cropped sub-image. Specifically, the larger the resolution of the target detection result, the more complete the target detection area and the better the quality of the target detection result. Therefore, the resolution sizes of the target detection results with the same coordinates in the video frame image to be processed are first compared. If the target recognition or subsequent analysis and processing has high requirements for image quality, the target detection result with the largest resolution is retained. Secondly, if the resolution sizes of the target detection results are the same, the target detection result with the earlier image number of the cropped sub-image is retained. This embodiment does not limit the specific rules for deduplication based on the resolution size and image number, and can be flexibly set according to the actual needs of target recognition or subsequent analysis and processing.

[0085] S280: Perform target recognition according to the target detection result after deduplication processing.

[0086] by Figure 4 For example, target detection is performed on the four first cropped sub-images and five second cropped sub-images. Targets are detected in the first cropped sub-image (3), the second cropped sub-image (5), and the second cropped sub-image (7). Then, for the three target detection results, the image number of the cropped sub-image, the coordinates in the cropped sub-image, and the coordinate transformation relationship of the cropped sub-image are determined respectively. For the three target detection results, coordinate transformation is performed according to the coordinates in the cropped sub-image and the coordinate transformation relationship of the cropped sub-image, respectively, to obtain the coordinates of the three target detection results in the video frame image to be processed. After the transformation, it can be seen that the target detection result corresponding to the first cropped sub-image (3) is the same as the target detection result of the second cropped sub-image (7) after transformation. It is determined that the two target detection results correspond to the same target, and only one of them can be retained.

[0087] The technical solution of the embodiment of the present invention determines the size of the first cropped sub-image according to the resolution of the video frame image and the resolution limit of the video analysis device, and performs the first cropping of the video frame image to be processed according to the size of the first cropped sub-image to obtain an equal number of first cropped sub-images, thereby ensuring that the cropped image can meet the resolution limit of the video analysis device and improving the adaptability of the video analysis device to high-resolution images. Then, the cropping boundary during the first cropping is determined, and the video frame image to be processed is cropped for the second time according to the cropping boundary to obtain a second cropped sub-image. The image area composed of each second cropped sub-image can completely cover the cropping boundary, thereby preventing the target from being missed and ensuring the target detection rate and the accuracy of target detection. For each first cropped sub-image and each second cropped sub-image, determine the image number and the coordinate transformation relationship between the cropped sub-image and the video frame image to be processed, after performing target detection on each first cropped sub-image and each second cropped sub-image, perform coordinate transformation on each target detection result, when there are at least two target detection results with the same coordinates after transformation, deduplicate the target detection results with the same coordinates after transformation, and then perform target recognition, thereby avoiding performance waste on subsequent processes caused by duplicate targets and ensuring the performance of subsequent target recognition.

[0088] Embodiment 3

[0089] Figure 6 FIG. 1 is a schematic diagram of a video frame image processing device provided in Embodiment 3 of the present invention. Figure 6 As shown, the device includes: a cropping boundary determination module 310 , a primary cropping module 320 , a secondary cropping module 330 and a target recognition module 340 .

[0090] A cropping boundary determination module 310, configured to determine a cropping boundary for a first cropping according to a resolution of a video frame image and a resolution limit of a video analysis device;

[0091] A first cropping module 320, configured to perform a first cropping on the video frame image to be processed according to a first cropping boundary to obtain at least two first cropped sub-images;

[0092] A secondary cropping module 330 is used to perform secondary cropping on the video frame image to be processed according to the cropping boundary during the first cropping of the video frame image to be processed, so as to obtain at least two secondary cropped sub-images, wherein the cropping boundary during the first cropping is located within the secondary cropped sub-images;

[0093] The target recognition module 340 is used to perform target recognition according to each first cropped sub-image and each second cropped sub-image.

[0094] The technical solution of the embodiment of the present invention determines the cropping boundary of the first cropping according to the resolution of the video frame image and the resolution limit of the video analysis device, performs the first cropping on the processed video frame image according to the cropping boundary of the first cropping, obtains at least two first cropped sub-images, and then performs the second cropping on the processed video frame image according to the cropping boundary when the processed video frame image is first cropped, obtains at least two second cropped sub-images, the cropping boundary is located within the at least two second cropped sub-images, and performs target recognition according to each first cropped sub-image and each second cropped sub-image. The problem in the prior art that the method of processing high-resolution images through pixel sampling will cause the quality of the video frame image to be reduced, thereby causing the target to be not recognized or the recognition accuracy to be reduced, and the problem that the method of processing high-resolution images through image segmentation is easy to cause the target to be damaged or lost due to cropping is solved, and the applicability of the video analysis device to high-resolution images is improved while ensuring the quality of the video frame image.

[0095] Based on the above embodiment, the sizes of the first cropped sub-images after the first cropping are the same and / or different;

[0096] The sizes of the secondary cropped sub-images after secondary cropping are the same and / or different;

[0097] The size of the secondary cropped sub-image meets the resolution limit of the video analysis device;

[0098] The size of the second cropped sub-image is the same as and / or different from the size of the first cropped sub-image;

[0099] A distance between the cropping boundary and a side of the secondary cropping sub-image containing the cropping boundary and parallel to the cropping boundary is greater than or equal to a standard target size.

[0100] Based on the above embodiment, the clipping boundary determination module 310 includes:

[0101] An equal division number determination unit, used to determine the equal division number of the video frame image according to the ratio of the resolution of the video frame image to the resolution limit of the video analysis device;

[0102] The cropping boundary determination unit is used to determine the cropping boundary of the first cropping according to the size of the video frame image and the number of equal divisions.

[0103] Based on the above embodiment, if the sizes of the first cropped sub-images after the first cropping are different, the length size of the first cropped sub-images after the first cropping meets the length resolution limit of the video analysis device, and the height size of the first cropped sub-images after the first cropping meets the height resolution limit of the video analysis device.

[0104] Based on the above embodiment, the secondary cropping module 330 includes:

[0105] A cropping line segment and cropping intersection determination unit, used to determine the cropping line segments and cropping intersections according to the cropping boundaries when the video frame image to be processed is cropped for the first time;

[0106] A secondary cropping unit, used for performing secondary cropping on the video frame image to be processed according to the cropping line segments and the cropping intersection points, to obtain at least two secondary cropped sub-images;

[0107] The cropping line segment is the center line of the secondary cropping sub-image including the cropping line segment, and the cropping intersection is the center point of the secondary cropping sub-image including the cropping intersection.

[0108] Based on the above embodiment, the target recognition module 340 includes:

[0109] The target detection unit is used to perform target detection according to each first cropped sub-image and each second cropped sub-image to obtain a first target detection result and a second target detection result;

[0110] The target detection result comparison unit is used to compare the first target detection result with the second target detection result to determine the comparison result;

[0111] A deduplication unit, for deduplicating the first target detection result and the second target detection result corresponding to the repeated target if it is determined according to the comparison result that there is a repeated target;

[0112] The target recognition unit is used to perform target recognition based on the target detection result after deduplication processing.

[0113] Based on the above embodiment, the target detection result comparison unit is specifically used for:

[0114] For the first target detection result and the second target detection result, respectively determine the image number of the cropped sub-image, the coordinates in the cropped sub-image, and the coordinate conversion relationship between the cropped sub-image corresponding to the cropped sub-image and the video frame image to be processed;

[0115] According to the coordinates of the first target detection result in the first cropped sub-image and the coordinate transformation relationship between the cropped sub-image corresponding to the first cropped sub-image and the video frame image to be processed, the first target detection result is transformed to obtain the coordinates of the first target detection result in the video frame image to be processed;

[0116] According to the coordinates of the secondary target detection result in the secondary cropped sub-image, and the coordinate transformation relationship between the cropped sub-image corresponding to the secondary cropped sub-image and the video frame image to be processed, the coordinates of the secondary target detection result in the video frame image to be processed are transformed;

[0117] The coordinates of the first target detection result in the video frame image to be processed are compared with the coordinates of the second target detection result in the video frame image to be processed to obtain a comparison result.

[0118] The video frame image processing device provided in the embodiment of the present invention can execute the video frame image processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0119] Embodiment 4

[0120] Figure 7 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0121] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0122] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a method for processing video frame images.

[0124] In some embodiments, the method for processing video frame images may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for processing video frame images described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for processing video frame images in any other appropriate manner (e.g., by means of firmware).

[0125] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0127] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0130] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0131] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0132] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for processing a video frame image, characterized in that: include: Determine a cropping boundary for the first cropping according to the resolution of the video frame image and the resolution limit of the video analysis device; Performing a first cropping on the video frame image to be processed according to the cropping boundary of the first cropping to obtain at least two first cropped sub-images; Performing secondary cropping on the video frame image to be processed according to the cropping boundary during the first cropping of the video frame image to be processed, to obtain at least two secondary cropped sub-images, wherein the cropping boundary during the first cropping is located within the secondary cropped sub-images; Target recognition is performed based on each first-cropped sub-image and each second-cropped sub-image.

2. The method according to claim 1, characterized in that The sizes of the first cropped sub-images after the first cropping are the same and / or different; The sizes of the secondary cropped sub-images after secondary cropping are the same and / or different; The size of the secondary cropped sub-image meets the resolution limit of the video analysis device; The size of the second cropped sub-image is the same as and / or different from the size of the first cropped sub-image; A distance between the cropping boundary and a side of the secondary cropping sub-image containing the cropping boundary and parallel to the cropping boundary is greater than or equal to a standard target size.

3. The method according to claim 2, characterized in that If the sizes of the first cropped sub-images after the first cropping are the same, the cropping boundary of the first cropping is determined according to the resolution of the video frame image and the resolution limit of the video analysis device, including: Determine the number of equal divisions of the video frame image according to a ratio of the resolution of the video frame image to a resolution limit of the video analysis device; A cropping boundary for the first cropping is determined according to the size of the video frame image and the number of equal divisions.

4. The method according to claim 2, characterized in that: If the sizes of the first cropped sub-images after the first cropping are different, the length size of the first cropped sub-images after the first cropping meets the length resolution limit of the video analysis device, and the height size of the first cropped sub-images after the first cropping meets the height resolution limit of the video analysis device.

5. The method according to claim 1, characterized in that: According to the cropping boundary when the video frame image to be processed is cropped for the first time, the video frame image to be processed is cropped for the second time to obtain at least two second-cropped sub-images, including: Determine the cropping line segments and cropping intersections according to the cropping boundaries when the video frame image to be processed is cropped for the first time; Performing secondary cropping on the video frame image to be processed according to the cropping line segments and the cropping intersection points to obtain at least two secondary cropped sub-images; The cropping line segment is the center line of the secondary cropping sub-image including the cropping line segment, and the cropping intersection is the center point of the secondary cropping sub-image including the cropping intersection.

6. The method according to claim 1, characterized in that According to each first cropped sub-image and each second cropped sub-image, target recognition is performed, including: Performing target detection according to each first cropped sub-image and each second cropped sub-image to obtain a first target detection result and a second target detection result; Compare the first target detection result and the second target detection result to determine the comparison result; If it is determined that there are duplicate targets according to the comparison results, the first target detection results and the second target detection results corresponding to the duplicate targets are deduplicated; Target recognition is performed based on the target detection results after deduplication processing.

7. The method according to claim 6, characterized in that Compare the first target detection result and the second target detection result to determine the comparison result, including: For the first target detection result and the second target detection result, respectively determine the image number of the cropped sub-image, the coordinates in the cropped sub-image, and the coordinate conversion relationship between the cropped sub-image corresponding to the cropped sub-image and the video frame image to be processed; According to the coordinates of the first target detection result in the first cropped sub-image and the coordinate transformation relationship between the cropped sub-image corresponding to the first cropped sub-image and the video frame image to be processed, the first target detection result is transformed to obtain the coordinates of the first target detection result in the video frame image to be processed; According to the coordinates of the secondary target detection result in the secondary cropped sub-image, and the coordinate transformation relationship between the cropped sub-image corresponding to the secondary cropped sub-image and the video frame image to be processed, the coordinates of the secondary target detection result in the video frame image to be processed are transformed; The coordinates of the first target detection result in the video frame image to be processed are compared with the coordinates of the second target detection result in the video frame image to be processed to obtain a comparison result.

8. A video frame image processing device, characterized in that: include: A cropping boundary determination module, used to determine a cropping boundary for the first cropping according to the resolution of the video frame image and the resolution limit of the video analysis device; A first cropping module, used for performing a first cropping on the video frame image to be processed according to a first cropping boundary, to obtain at least two first cropped sub-images; A secondary cropping module, used for performing secondary cropping on the video frame image to be processed according to the cropping boundary during the first cropping of the video frame image to be processed, so as to obtain at least two secondary cropped sub-images, wherein the cropping boundary during the first cropping is located within the secondary cropped sub-images; The target recognition module is used to perform target recognition based on each first cropped sub-image and each second cropped sub-image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for processing a video frame image as described in any one of claims 1 to 7 is implemented.

10. A storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the video frame image processing method as described in any one of claims 1-7 when executed by a computer processor.