Target highlighting method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a target highlighting method and device, electronic equipment and a storage medium, and relates to the technical field of machine vision, and the method comprises the steps: carrying out the target recognition of a to-be-processed image frame in a to-be-processed video, and obtaining a to-be-utilized detection frame and corresponding first mask information; determining a cache frame different from the interested target represented by each to-be-utilized detection frame from the cache set as a to-be-processed cache frame; for each to-be-processed cache frame, if mask information matched with the to-be-processed cache frame does not exist in the first mask information, performing image segmentation on an image region corresponding to the to-be-processed cache frame in the to-be-processed image frame to obtain second mask information; and in the to-be-processed image frame, highlighting the target image area represented by the obtained first mask information and the second mask information. According to the scheme, the stability of the video picture after the target is highlighted can be improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a target highlighting method, device, electronic device, and storage medium. Background Art

[0002] Target highlighting refers to identifying the location of an object of interest in an image and then enhancing the image region where the object of interest is located to make the object of interest more visually obvious. For example, the image region where the object of interest is located can be pseudo-colored.

[0003] However, when it is necessary to highlight the target in each image frame in the video, if the environment when collecting the video is complex, it may cause the targets of interest in some discontinuous image frames in the video to be missed. For example, the targets of interest in the first and second image frames in the video are recognized, while the targets of interest in the third and fourth image frames are not recognized, and the target of interest in the fifth image frame is recognized. At this time, if the target highlighting processing is performed on such a video, the picture will be unstable when the video is played, and the picture will flicker, which greatly affects the viewing experience of the video.

[0004] Therefore, how to improve the stability of the video image after the target is highlighted is an urgent problem to be solved. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device, and storage medium for highlighting an object, so as to improve the stability of a video image after highlighting the object. The specific technical solution is as follows:

[0006] The present invention provides a method for highlighting a target, the method comprising:

[0007] Performing target recognition on an image frame to be processed in a video to be processed, obtaining a detection frame to be used and corresponding first mask information of an object of interest in the image frame to be processed; wherein the first mask information corresponding to the detection frame of the object of interest represents: a target image area occupied by the object of interest within the detection frame;

[0008] Determining, from a current cache set, cache frames that are different from the objects of interest represented by each detection frame to be used as cache frames to be processed; wherein the cache frames in the current cache set include: detection frames representing different objects of interest identified in a first number of historical image frames adjacent to the image frame to be processed;

[0009] Performing image segmentation on an image region corresponding to each buffer frame to be processed in the image frame to be processed, and obtaining second mask information representing a target image region occupied by an object of interest within the image region;

[0010] In the image frame to be processed, the target image region represented by the obtained first mask information and the second mask information is highlighted.

[0011] In one embodiment, determining, from the current cache set, cache frames that are different from the objects of interest represented by each detection frame to be used as cache frames to be processed includes:

[0012] For each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a candidate cache frame;

[0013] For each candidate cache frame, if there is no mask information matching the candidate cache frame in the first mask information, the candidate cache frame is determined as the cache frame to be processed; wherein the matching mask information and the category of the target of interest represented by the cache frame are consistent, and the portion of the target image area represented by the mask information within the cache frame is greater than a first threshold.

[0014] In one embodiment, determining, from the current cache set, cache frames that are different from the objects of interest represented by each detection frame to be used as cache frames to be processed includes:

[0015] For each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a cache frame to be processed.

[0016] In one embodiment, the method further comprises:

[0017] For each cache frame in the current cache set, if the cache frame meets any of the following conditions, the cache frame will be deleted from the current cache set:

[0018] Among the obtained detection frames to be used, there is a detection frame to be used whose intersection-over-union ratio with the cache frame is greater than a preset intersection-over-union ratio;

[0019] The cache frame and the detection frames identified in the first number of image frames to be compared represent different objects of interest; wherein the image frames to be compared include the image frame to be processed and adjacent historical image frames;

[0020] The obtained detection frame to be used is added as a cache frame to the current cache set.

[0021] In one embodiment, the intersection-over-union (IoU) ratio between the detection frame and the cache frame is greater than a preset IoU ratio and satisfies at least one of the following conditions:

[0022] The intersection-over-union ratio of the image area represented by the detection frame and the cache frame is greater than a second threshold;

[0023] The intersection-over-union ratio of the target image area represented by the first mask information corresponding to the detection frame and the cache frame is greater than a third threshold;

[0024] An intersection-over-union ratio of the extended image areas corresponding to the detection frame and the cache frame is greater than a fourth threshold.

[0025] In one embodiment, performing image segmentation on an image region corresponding to each buffer frame to be processed in the image frame to be processed to obtain second mask information representing a target image region occupied by an object of interest in the image region includes:

[0026] For the image area corresponding to each cache frame to be processed in the image frame to be processed, the image area is segmented using the maximum entropy threshold segmentation algorithm based on the gradient of each pixel point in the image area to obtain second mask information representing the target image area occupied by the target of interest in the image area.

[0027] In one embodiment, the performing target recognition on the image frame to be processed in the video to be processed to obtain a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed includes:

[0028] Performing target recognition on the image frame to be processed in the video to be processed, and obtaining an initial detection frame of the target of interest in the image frame to be processed and corresponding first mask information;

[0029] Detection frames that meet a preset deletion condition are deleted from the obtained initial detection frames, and the remaining detection frames are used as detection frames to be used, thereby obtaining the detection frame to be used of the target of interest in the image frame to be processed and the corresponding first mask information.

[0030] In one embodiment, the target of interest is an animal, and the preset deletion condition includes at least one of the following:

[0031] The ratio of the area represented by the first mask information corresponding to the detection frame to the area represented by the detection frame is less than a fifth threshold;

[0032] The ratio of the short side to the long side of the detection frame is less than a sixth threshold;

[0033] The average level of infrared radiation intensity of the image area represented by the first mask information corresponding to the detection frame is less than a seventh threshold.

[0034] In one embodiment, before performing target recognition on the image frame to be processed in the video to be processed and obtaining a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed, the method further includes:

[0035] Determining state information corresponding to the image frame to be processed; wherein the state information corresponding to an image frame indicates whether the image acquisition device is in motion when acquiring the image frame;

[0036] determining, based on the state information corresponding to the image frame to be processed, whether the image acquisition device is in a stable state when acquiring the image frame to be processed;

[0037] The method of performing target recognition on the image frame to be processed in the video to be processed and obtaining a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed includes:

[0038] When it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed, target recognition is performed on the image frame to be processed in the video to be processed to obtain a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed.

[0039] In one embodiment, determining whether the image acquisition device is in a stable state when acquiring the image frame to be processed based on the state information corresponding to the image frame to be processed includes:

[0040] In a case where the image acquisition device was in a stable state when acquiring the previous image frame of the image frame to be processed, if the state information corresponding to the image frame to be processed and a second number of adjacent historical image frames all indicate a motion state, then it is determined that the image acquisition device was in an unstable state when acquiring the image frame to be processed; if the state information corresponding to any one of the image frame to be processed and the second number of adjacent historical image frames indicates a non-motion state, then it is determined that the image acquisition device was in a stable state when acquiring the image frame to be processed;

[0041] In the case that the image acquisition device is in an unstable state when acquiring the previous image frame of the image frame to be processed, if the status information corresponding to the image frame to be processed and the second number of adjacent historical image frames all indicate a non-motion state, it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed; if the status information corresponding to any image frame among the image frame to be processed and the second number of adjacent historical image frames indicates a motion state, it is determined that the image acquisition device is in an unstable state when acquiring the image frame to be processed.

[0042] In one embodiment, determining the status information corresponding to the image frame to be processed includes:

[0043] Acquiring multiple angular velocities collected by the gyroscope of the image acquisition device for each acquisition direction during a period from the acquisition of the previous image frame to the acquisition of the image frame to be processed;

[0044] For each acquisition direction, calculating the sum of the angular velocities acquired by the gyroscope for the acquisition direction, and obtaining first motion information of each acquisition direction corresponding to the image frame to be processed;

[0045] For each acquisition direction, calculating the range of the angular velocity acquired by the gyroscope for the acquisition direction, and calculating the sum of the obtained ranges of each acquisition direction to obtain second motion information corresponding to the image frame to be processed;

[0046] Based on the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed, the state information corresponding to the image frame to be processed is determined.

[0047] In one embodiment, determining the state information corresponding to the image frame to be processed based on the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed includes:

[0048] The first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed are input into a pre-trained decision model to obtain the state information corresponding to the image frame to be processed output by the decision model; wherein the decision model is: a model trained based on the first motion information of each acquisition direction corresponding to the sample image acquired by the sample image acquisition device and the second motion information corresponding to the sample image, and the actual state of the sample image acquisition device when acquiring the sample image.

[0049] The present application also provides a target highlighting device, comprising:

[0050] a target recognition module configured to perform target recognition on an image frame to be processed in a video to be processed, and obtain a detection frame to be used and corresponding first mask information of an object of interest in the image frame to be processed; wherein the first mask information corresponding to the detection frame of the object of interest represents: a target image area occupied by the object of interest within the detection frame;

[0051] a buffer frame determination module for processing, configured to determine, from a current buffer set, buffer frames that are different from the objects of interest represented by each detection frame to be used, as buffer frames to be processed; wherein the buffer frames in the current buffer set include detection frames representing different objects of interest identified in a first number of historical image frames adjacent to the image frame to be processed;

[0052] An image segmentation module is used to perform image segmentation on an image region corresponding to each buffer frame to be processed in the image frame to be processed, and obtain second mask information representing a target image region occupied by an object of interest in the image region;

[0053] The highlighting module is used to highlight the target image area represented by the obtained first mask information and the second mask information in the image frame to be processed.

[0054] In one embodiment, the module for determining a cache frame to be processed includes:

[0055] a candidate cache frame determination submodule, configured to determine, for each cache frame in the current cache set, a cache frame as a candidate cache frame if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio;

[0056] A submodule for determining a cache frame to be processed is configured to determine, for each candidate cache frame, if there is no mask information matching the candidate cache frame in the first mask information, the candidate cache frame as the cache frame to be processed; wherein the matching mask information is consistent with the category of the target of interest represented by the cache frame, and the portion of the target image area represented by the mask information within the cache frame is greater than a first threshold.

[0057] In one embodiment, the module for determining the cache frame to be processed is specifically configured to:

[0058] For each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a cache frame to be processed.

[0059] In one embodiment, the apparatus further comprises:

[0060] a cache frame deletion module configured to delete each cache frame in the current cache set from the current cache set if the cache frame satisfies any of the following conditions: a cache frame exists among the obtained detection frames to be used whose intersection-over-union (IoU) with the cache frame is greater than a preset IoU; and the cache frame and the detection frames identified in the first number of image frames to be compared represent different objects of interest; wherein the image frames to be compared include the image frame to be processed and adjacent historical image frames;

[0061] The cache frame adding module is used to add the obtained detection frame to be used as a cache frame to the current cache set.

[0062] In one embodiment, the intersection-over-union (IoU) ratio between the detection frame and the cache frame is greater than a preset IoU ratio and satisfies at least one of the following conditions:

[0063] The intersection-and-union ratio of the image areas represented by the detection frame and the cache frame is greater than the second threshold; the intersection-and-union ratio of the target image areas represented by the first mask information corresponding to the detection frame and the cache frame is greater than the third threshold; the intersection-and-union ratio of the extended image areas corresponding to the detection frame and the cache frame is greater than the fourth threshold.

[0064] In one embodiment, the image segmentation module is specifically configured to:

[0065] For the image area corresponding to each cache frame to be processed in the image frame to be processed, the image area is segmented using the maximum entropy threshold segmentation algorithm based on the gradient of each pixel point in the image area to obtain second mask information representing the target image area occupied by the target of interest in the image area.

[0066] In one embodiment, the target recognition module includes:

[0067] The target recognition submodule is used to perform target recognition on the image frame to be processed in the video to be processed, and obtain an initial detection frame of the target of interest in the image frame to be processed and corresponding first mask information;

[0068] The screening submodule is used to delete the detection frames that meet the preset deletion conditions from the obtained initial detection frames, and use the remaining detection frames as the detection frames to be used, to obtain the detection frames to be used of the target of interest in the image frame to be processed and the corresponding first mask information.

[0069] In one embodiment, the target of interest is an animal, and the preset deletion condition includes at least one of the following:

[0070] The ratio of the area represented by the first mask information corresponding to the detection frame to the area represented by the detection frame is less than the fifth threshold; the ratio of the short side to the long side of the detection frame is less than the sixth threshold; the average level of infrared radiation intensity of the image area represented by the first mask information corresponding to the detection frame is less than the seventh threshold.

[0071] In one embodiment, the apparatus further comprises:

[0072] a state information determination module configured to determine state information corresponding to the image frame to be processed before the target recognition module performs target recognition on the image frame to be processed in the video to be processed and obtains a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed; wherein the state information corresponding to an image frame indicates whether the image acquisition device was in motion when the image frame was acquired;

[0073] a stable state judgment module, configured to determine whether the image acquisition device is in a stable state when acquiring the image frame to be processed based on state information corresponding to the image frame to be processed;

[0074] The target recognition module is specifically used to perform target recognition on the image frame to be processed in the video to be processed when it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed, and obtain a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed.

[0075] In one embodiment, the stable state judgment module is specifically configured to:

[0076] In a case where the image acquisition device was in a stable state when acquiring the previous image frame of the image frame to be processed, if the state information corresponding to the image frame to be processed and a second number of adjacent historical image frames all indicate a motion state, then it is determined that the image acquisition device was in an unstable state when acquiring the image frame to be processed; if the state information corresponding to any one of the image frame to be processed and the second number of adjacent historical image frames indicates a non-motion state, then it is determined that the image acquisition device was in a stable state when acquiring the image frame to be processed;

[0077] In the case that the image acquisition device is in an unstable state when acquiring the previous image frame of the image frame to be processed, if the status information corresponding to the image frame to be processed and the second number of adjacent historical image frames all indicate a non-motion state, it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed; if the status information corresponding to any image frame among the image frame to be processed and the second number of adjacent historical image frames indicates a motion state, it is determined that the image acquisition device is in an unstable state when acquiring the image frame to be processed.

[0078] In one embodiment, the status information determination module includes:

[0079] an angular velocity acquisition submodule, configured to acquire a plurality of angular velocities acquired by the gyroscope of the image acquisition device for each acquisition direction during a period from the acquisition of the previous image frame to the acquisition of the image frame to be processed;

[0080] a first calculation submodule, configured to calculate, for each acquisition direction, the sum of the angular velocities acquired by the gyroscope in the acquisition direction, and obtain first motion information in each acquisition direction corresponding to the image frame to be processed;

[0081] a second calculation submodule, configured to calculate, for each acquisition direction, a range of angular velocities acquired by the gyroscope in the acquisition direction, and calculate a sum of the obtained ranges for each acquisition direction to obtain second motion information corresponding to the image frame to be processed;

[0082] The state information determination submodule is configured to determine the state information corresponding to the image frame to be processed based on the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed.

[0083] In one embodiment, the state information determination submodule is specifically configured to:

[0084] The first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed are input into a pre-trained decision model to obtain the state information corresponding to the image frame to be processed output by the decision model; wherein the decision model is: a model trained based on the first motion information of each acquisition direction corresponding to the sample image acquired by the sample image acquisition device and the second motion information corresponding to the sample image, and the actual state of the sample image acquisition device when acquiring the sample image.

[0085] An embodiment of the present application further provides an electronic device, including:

[0086] The memory is used to store computer programs; the processor is used to implement any of the above-mentioned target highlighting methods when executing the programs stored in the memory.

[0087] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned target highlighting methods is implemented.

[0088] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described target highlighting methods.

[0089] Beneficial effects of the embodiments of the present application:

[0090] The target highlighting method for an image provided by an embodiment of the present application first performs target recognition on an image frame to be processed in a video to be processed, and obtains a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed; then, from adjacent historical image frames, a cache frame that is different from the target of interest represented by the detection frame in the image frame to be processed is determined as the cache frame to be processed. Since the position change of the same target of interest in adjacent image frames will not be too large, for the cache frame to be processed, if there is no mask information matching the cache frame to be processed in the first mask information, the matching mask information and the category of the target of interest represented by the cache frame to be processed are consistent, and the part of the target image area represented by the mask information within the cache frame to be processed is greater than a first threshold, it indicates that there is a high probability that the target of interest is missed in the image area corresponding to the cache frame to be processed in the image frame to be processed, and then image segmentation is performed on the image area in the image frame to be processed to obtain second mask information representing the target image area occupied by the target of interest in the image area. It can be seen that this scheme can detect image areas that may be missed in the image frame to be processed, and perform further image segmentation processing on the image area, thereby reducing the missed detection of targets of interest in the video. Therefore, in the image frame to be processed, the target image area represented by the first mask information and the second mask information is finally highlighted, which can improve the stability of the video picture after the target is highlighted and improve the viewing experience of the video.

[0091] Of course, it is not necessary to achieve all the advantages described above at the same time when implementing any product or method of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0093] Figure 1 A schematic diagram of a first flow chart of a target highlighting method provided in an embodiment of the present application;

[0094] Figure 2 A schematic diagram of the image segmentation process in the target highlighting method provided in an embodiment of the present application;

[0095] Figure 3 A second flow chart of the target highlighting method provided in an embodiment of the present application;

[0096] Figure 4 A third flow chart of the target highlighting method provided in an embodiment of the present application;

[0097] Figure 5 A fourth flow chart of the target highlighting method provided in an embodiment of the present application;

[0098] Figure 6 A schematic diagram of the structure of a target highlighting device provided in an embodiment of the present application;

[0099] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0100] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of this application.

[0101] The task of target detection is to find all targets of interest in the image and mark the location of the targets of interest. This process includes two steps: classification and positioning. The role of classification is to identify the category of the target of interest, and positioning is to determine the specific location of the target of interest in the image, usually the minimum enclosing rectangle (hereinafter referred to as the detection box). On the other hand, target segmentation is based on target detection. It can not only output the category and location of the target of interest, but also segment the target of interest in the detection box, and output the mask information representing the image area occupied by the target of interest in the detection box, so as to cut out the target of interest from the image. Furthermore, on the basis of target segmentation, the image area represented by the mask information can be highlighted, which can make the target of interest more visually obvious.

[0102] However, there may be targets of interest in the scene whose shapes change rapidly (such as running prey). The rapid changes in shape during the movement process, coupled with the occlusion of interference objects such as complex scene backgrounds, are prone to misidentification and missed identification. If such videos are subjected to target highlighting processing, the picture will be unstable when the video is played, and the picture will flicker, which greatly affects the viewing experience of the video.

[0103] To improve the stability of the video image after the target is highlighted and enhance the viewing experience of the video, the embodiments of the present application provide a target highlighting method, device, electronic device, and storage medium. The method can be used in electronic devices with data processing capabilities. In specific scenarios, the method can be used in image acquisition devices such as smart cameras and thermal imagers. The method includes the following steps:

[0104] Performing target recognition on an image frame to be processed in the video to be processed to obtain a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed; wherein the first mask information corresponding to the detection frame of the target of interest represents: a target image area occupied by the target of interest within the detection frame;

[0105] Determining, from a current cache set, cache frames that are different from the objects of interest represented by each detection frame to be used as cache frames to be processed; wherein the cache frames in the current cache set include: detection frames representing different objects of interest identified in a first number of historical image frames adjacent to the image frame to be processed;

[0106] Performing image segmentation on an image region corresponding to each buffer frame to be processed in the image frame to be processed, and obtaining second mask information representing a target image region occupied by an object of interest in the image region;

[0107] In the image frame to be processed, the target image region represented by the obtained first mask information and the second mask information is highlighted.

[0108] In this embodiment, it is possible to detect image areas that may be missed in the image frame to be processed, and further image segmentation processing is performed on the image area, thereby reducing the missed detection of targets of interest in the video. Therefore, in the image frame to be processed, the target image area represented by the first mask information and the second mask information is finally highlighted, which can improve the stability of the video picture after the target is highlighted and improve the viewing experience of the video.

[0109] The target highlighting method provided by the embodiment of the present application is introduced below with reference to the accompanying drawings. Figure 1 As shown, the method includes the following steps:

[0110] S101, performing target recognition on an image frame to be processed in a video to be processed, and obtaining a detection frame to be used and corresponding first mask information of an object of interest in the image frame to be processed;

[0111] The first mask information corresponding to the detection frame of the object of interest represents the target image area occupied by the object of interest within the detection frame. For example, the first mask information may represent whether each pixel within the detection frame belongs to the object of interest.

[0112] The video to be processed in this embodiment can be a video captured in real time by an image acquisition device, and the image frame to be processed can be an image frame currently being captured in real time by the image acquisition device. In this case, each time the image acquisition device captures an image frame, the steps of this embodiment can be executed to highlight the target in that image frame. The video to be processed can also be a video captured and saved historically, and the image frame to be processed can be any frame in the video to be processed. In this case, each image frame in the video to be processed can be sequentially highlighted using this embodiment.

[0113] If the image frames captured by the image acquisition device are visible light images, each image frame in the video to be processed can be a visible light image. If the image frames captured by the image acquisition device are thermal images, each image frame in the video to be processed can be a thermal image. Each pixel in a thermal image typically contains 8 bits of data. In scenes that require both daytime and nighttime viewing, highlighting targets in thermal images can be more effective.

[0114] The target of interest in this embodiment can be set according to actual conditions. For example, when it is necessary to detect animals in a forest scene, the target of interest can include multiple animals, such as tigers, bears, monkeys, etc.

[0115] Target recognition of the image frames to be processed in the video to be processed can be achieved using a pre-trained target segmentation model, for example, the target segmentation model can be a U-Net (U-net) model, a VGG (Visual Geometry Group) neural network model, etc.

[0116] In one case, the target segmentation model can be used to detect the detection frames of all targets of interest in the image frame to be processed as the detection frames to be used. In another case, due to the detection accuracy of the target segmentation model, there may be some false detection frames in the detection frames detected by the target segmentation model. For example, when detecting an image of a forest scene, if the target of interest is an animal, the detection frame detected by the target segmentation model may include detection frames of objects such as tree trunks, leaves, and stones (i.e., false detection frames). Therefore, the detected detection frames can be screened to eliminate false detection frames and obtain detection frames that actually contain the target of interest as the detection frames to be used. The specific screening process can be seen in step S302 below.

[0117] S102, determining, from the current cache set, cache frames that are different from the objects of interest represented by each detection frame to be used as cache frames to be processed;

[0118] The cache frames in the current cache set include: detection frames representing different objects of interest identified in a first number of historical image frames adjacent to the image frame to be processed. The cache frames in the cache set are actually also detection frames.

[0119] "The cache frame and the detection frame to be used represent different objects of interest" means that the cache frame and the detection frame to be used do not represent the same object of interest. In this case, the categories of the objects of interest represented by the cache frame and the detection frame to be used may be the same or different. Correspondingly, "the cache frame and the detection frame to be used represent the same object of interest" means that the cache frame and the detection frame to be used represent the same object of interest. In this case, the categories of the objects of interest represented by the cache frame and the detection frame to be used are also the same.

[0120] Because the detection of the same target of interest in consecutive image frames is unstable, and when the target of interest is an animal, its motion state is variable, and its posture and position angle change during movement. For example, when the animal faces away from or facing the camera, the target recognition result will be inaccurate, resulting in flickering in the video. Therefore, in this embodiment, it is necessary to determine whether the detection frame in the historical image frame is the same as the target of interest to be represented by the detection frame, that is, to determine whether they represent the same target of interest.

[0121] In this step, for each cache frame in the current cache set, it can be determined whether the cache frame is different from each target of interest represented by the detection frame to be used. If they are different, the cache frame is used as a cache frame to be processed.

[0122] In this embodiment, it is possible to determine whether the detection frame to be used and the cache frame represent the same object of interest based on the intersection-over-union ratio. This embodiment provides two ways to determine the cache frame to be processed:

[0123] Method 1: For each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a cache frame to be processed.

[0124] The intersection-over-union (IoU) ratio between the detection frame and the cache frame is greater than the preset IoU ratio and satisfies at least one of the following conditions:

[0125] Condition 1: the intersection-over-union ratio of the image areas represented by the detection frame and the cache frame is greater than a second threshold;

[0126] Condition 2: the intersection-over-union ratio of the target image area represented by the first mask information corresponding to the detection frame and the cache frame is greater than a third threshold;

[0127] Condition three: an intersection-over-union ratio of the extended image areas corresponding to the detection frame and the cache frame is greater than a fourth threshold.

[0128] The extended image area corresponding to the detection frame or cache frame can be obtained by increasing the side length of the detection frame or cache frame by a specified ratio while maintaining the center point of the detection frame or cache frame unchanged. Due to the rapid movement of some objects of interest, it may be difficult to match the objects in the previous and next frames. In other words, it is difficult to determine the detection frames representing the same object of interest in the previous and next image frames using the intersection-over-union ratio. Therefore, in this embodiment, whether the detection frame and the extended image area corresponding to the cache frame represent the same object of interest is determined based on the intersection-over-union ratio, making it easier to match the same object of interest.

[0129] Method 2: Step A1, for each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a candidate cache frame;

[0130] Similarly, in this approach, if the intersection-over-union ratio between the detection frame and the cache frame is greater than the preset intersection-over-union ratio, it also satisfies at least one of the above three conditions.

[0131] Step A2: For each candidate cache frame, if there is no mask information matching the candidate cache frame in the first mask information, the candidate cache frame is determined as the cache frame to be processed; wherein the matching mask information and the category of the target of interest represented by the cache frame are consistent, and the portion of the target image area represented by the mask information within the cache frame is greater than the first threshold.

[0132] The first threshold can be set according to the size of the image frame to be processed and actual conditions. For example, the first threshold can be the product of the area of ​​the buffer frame to be processed and a preset ratio, and the preset ratio can be 0.3.

[0133] Specifically, we can first determine the image area corresponding to the alternative cache frame in the image frame to be processed, and determine the target image area represented by the first mask information, and then determine the overlapping part between the two image areas, and thus obtain the part of the target image area represented by the first mask information within the alternative cache frame.

[0134] Since the candidate cache frame is identified in historical image frames adjacent to the image frame to be processed, and the position of the target of interest in adjacent image frames generally does not change significantly, if the first mask information does not contain mask information matching the candidate cache frame, it indicates that the image region corresponding to the candidate cache frame in the image frame to be processed is likely to have missed detection of the target of interest, and the candidate cache frame can be used as the cache frame to be processed. If mask information matching the candidate cache frame exists, it indicates that the image region corresponding to the candidate cache frame has already been detected and is likely to have no missed detection. In this case, no further processing is required for the image region, and the candidate cache frame can be omitted as the cache frame to be processed.

[0135] For example, there are cache frame 1, cache frame 2 and cache frame 3 in the cache set; the detection frames to be used include: detection frame 4, detection frame 5 and detection frame 6. First, it is determined that the intersection-and-union ratio of cache frame 1 and detection frame 4 is greater than the preset intersection-and-union ratio, and the intersection-and-union ratios between cache frame 2 and cache frame 3 and each detection frame to be used are not greater than the preset intersection-and-union ratio. At this time, cache frame 2 and cache frame 3 can be determined as alternative cache frames. Furthermore, if there is mask information matching cache frame 2 in the first mask information, it indicates that cache frame 2 is a falsely detected cache frame, and finally only cache frame 3 can be used as the cache frame to be processed.

[0136] As can be seen, because Method 1 only determines whether the cache frame and the detection frame to be used represent the same target of interest by determining whether the intersection-over-union ratio (IoU) between the two frames is greater than a threshold, its accuracy is not high enough: if the IoU threshold (i.e., the second threshold, the third threshold, or the fourth threshold) is set too high, it is easy to miss a detection; if the threshold is set too low, it is easy to make a false detection. Therefore, based on Method 1, this method further screens the candidate cache frames based on the first mask information. During the specific implementation process, the IoU threshold can be set to a lower value, and then combined with step A2 of this method, false detections can be eliminated, thereby improving the accuracy of the determined cache frame to be processed.

[0137] In addition, illustratively, cache boxes in the cache set may be added or deleted in the following manner:

[0138] First, for the first image frame in the video to be processed, all the detection frames to be used in the image frame can be added to the cache set as cache frames. For each subsequent image frame, for each detection frame to be used in the image frame, it can be determined whether there is a cache frame in the cache set that represents the same target of interest as the detection frame to be used. If so, the detection frame to be used is used as the cache frame to replace the cache frame in the cache set that represents the same target of interest. If not, the detection frame to be used is directly added to the cache set as a cache frame. In addition, a count process can be performed on each cache frame in the cache set: the count value of the newly added cache frame is set to 0, and each time an image frame is processed, the count value of each cache frame is increased by 1 until the count of the cache frame reaches a first number, for example, if the first number is 5, the cache frame is deleted from the cache set.

[0139] That is, the method provided in the embodiment of the present application may also include a process of adding and deleting a cache frame:

[0140] Step B1: For each cache frame in the current cache set, if the cache frame meets any of the following conditions, the cache frame is deleted from the current cache set:

[0141] Condition 1: There is a detection frame to be used among the obtained detection frames to be used that represents the same object of interest as the cache frame.

[0142] In this way, for the same target of interest, the cache set can only contain the most recently detected detection frame representing the target of interest, which is more conducive to matching subsequent detection frames with cache frames, that is, determining whether the detection frame and the cache frame represent the same target of interest.

[0143] Condition 2: The cache frame and the detection frames identified in the first number of image frames to be compared represent different objects of interest; wherein the image frames to be compared include the image frames to be processed and adjacent historical image frames.

[0144] When a cache frame represents a different target of interest than the detection frame identified in the first number of image frames, it indicates that the acquisition time interval between the image frame to which the cache frame belongs and the image frame to be processed is large, and the change of the target of interest may also be large, making it difficult to accurately match it with the detection frame subsequently. Therefore, the cache frame can be deleted.

[0145] Step B2: Add the obtained detection frame to be used as a cache frame to the current cache set.

[0146] Specifically, when determining the cache frame to be processed using the first method described above, since the cache frame identical to the target of interest to be represented by the detection frame is also determined at the same time as the cache frame to be processed, for convenience, the process of adding and deleting cache frames can be performed after the cache frame to be processed is determined. When determining the cache frame to be processed using the second method described above, the process of adding and deleting cache frames can be performed after the candidate cache frames are determined.

[0147] Of course, the process of adding and deleting cache frames can also be performed before determining the cache frame to be processed, and after obtaining the detection frame to be used of the target of interest in the image frame to be processed. In this case, when executing the step of determining the cache frame to be processed, the detection frame to be used already exists in the current cache set. Therefore, when executing the step of determining the cache frame to be processed, there will be some repeated detection frame judgments, but in the end, a cache frame different from the target of interest represented by each detection frame to be used can be determined from the current cache set.

[0148] By adding and deleting cache frames in the cache set in the above manner, when any image frame needs to be processed, the cache set can contain detection frames representing different objects of interest identified in the first number of historical image frames adjacent to the image frame.

[0149] S103 , performing image segmentation on the image region corresponding to each buffer frame to be processed in the image frame to be processed, and obtaining second mask information representing the target image region occupied by the object of interest in the image region.

[0150] After determining the buffer frame to be processed, the image area corresponding to the buffer frame to be processed in the image frame to be processed can be determined. That is, according to the position and size of the buffer frame to be processed, the image area mapped by the buffer frame to be processed can be determined in the image frame to be processed. The image area corresponding to the buffer frame to be processed in the image frame to be processed is further segmented to obtain second mask information representing the target image area occupied by the target of interest within the image area, which can reduce the missed detection of the target of interest. Furthermore, after determining the image area in the image frame to be processed according to the position and size of the buffer frame to be processed, the determined image area can be expanded, and the expanded image area can be used as the image area corresponding to the buffer frame to be processed. The expansion is similar to the above. This can increase the area for image segmentation and is more conducive to completely segmenting the target of interest.

[0151] In one implementation, any image binarization algorithm, such as a fixed-threshold binarization algorithm, can be used to segment the image region. In another implementation, a maximum entropy threshold segmentation algorithm can also be used to segment the image region. The specific process will be described below.

[0152] S104 : In the image frame to be processed, highlight the target image region represented by the obtained first mask information and the second mask information.

[0153] The target image region represented by the first mask information and the second mask information in the image frame to be processed is, that is, the region to which the first mask information and the second mask information are mapped in the image frame to be processed.

[0154] In the image frame to be processed, the color of each pixel in the target image area represented by the first mask information and the second mask information may be replaced with a more eye-catching color, such as high-brightness red.

[0155] When the image frame to be processed is a thermal imaging image, pseudo color processing can be performed on the target image area represented by the first mask information and the second mask information, that is, each pixel in the target image area represented by the first mask information and the second mask information in the thermal imaging image is converted from gray to color.

[0156] In this embodiment, it is possible to detect image areas that may be missed in the image frame to be processed, and further image segmentation processing is performed on the image area, thereby reducing the missed detection of targets of interest in the video. Therefore, in the image frame to be processed, the target image area represented by the first mask information and the second mask information is finally highlighted, which can improve the stability of the video picture after the target is highlighted and improve the viewing experience of the video.

[0157] In one embodiment of the present application, image segmentation is performed on an image region corresponding to the buffer frame to be processed in the image frame to be processed to obtain second mask information representing a target image region occupied by an object of interest in the image region, including:

[0158] For the image area corresponding to each buffer frame to be processed in the image frame to be processed, the image area is segmented using the maximum entropy threshold segmentation algorithm based on the gradient of each pixel point in the image area to obtain the second mask information representing the target image area occupied by the target of interest in the image area. For example, Figure 2 As shown, the process may include the following steps:

[0159] S201, calculating the gradient of each pixel in the image area corresponding to the buffer frame to be processed in the image frame to be processed;

[0160] This embodiment does not limit the specific method of calculating the gradient of the pixel point. For example, the Sobel operator can be used to calculate the gradient of the pixel point.

[0161] S202, selecting a threshold from a preset threshold set as a current threshold;

[0162] S203, for each pixel in the image area, if the gradient of the pixel is greater than a current threshold, then the pixel is determined to be a pixel representing the object of interest in the image area; if the gradient of the pixel is not greater than the current threshold, then the pixel is determined to be a pixel representing the background in the image area;

[0163] S204, respectively calculating the information entropy of the pixel representing the target of interest and the information entropy of the pixel representing the background, and calculating the sum of the obtained information entropies as the information entropy to be used;

[0164] S205 , selecting the next threshold from the threshold set as the current threshold, and returning to execute S203 , until all thresholds in the threshold set are selected, and executing S206 .

[0165] S206, determining the maximum value of the information entropy to be used from the obtained information entropies to be used, and obtaining second mask information representing the target image area occupied by the target of interest in the image area based on the pixel point representing the target of interest determined when the maximum value of the information entropy to be used is obtained.

[0166] In this embodiment, based on the gradient of each pixel point in the image area corresponding to the cache frame to be processed in the image frame to be processed, the maximum entropy threshold segmentation algorithm is used to perform image segmentation on the image area. Compared with the segmentation method using a fixed threshold, this scheme can more accurately obtain the second mask information representing the target image area occupied by the target of interest in the image area, thereby further improving the stability of the video image after the target is highlighted.

[0167] In one embodiment of the present application, Figure 3 As shown, the target highlighting method may include the following steps:

[0168] S301, performing target recognition on an image frame to be processed in a video to be processed, and obtaining an initial detection frame of an object of interest in the image frame to be processed and corresponding first mask information;

[0169] This step can be implemented using a pre-trained target segmentation model. Since target detection and target segmentation are probabilistic prediction problems, false detections are inevitable, so false detection filtering is also required.

[0170] S302 , deleting the detection frames that meet the preset deletion condition from the obtained initial detection frames, and using the remaining detection frames as the detection frames to be used, to obtain the detection frames to be used of the target of interest in the image frame to be processed and the corresponding first mask information.

[0171] When the target of interest is an animal, the preset deletion condition includes at least one of the following:

[0172] The first item is that the ratio of the area represented by the first mask information corresponding to the detection frame to the area represented by the detection frame is less than a fifth threshold;

[0173] The second item is that the ratio of the short side to the long side of the detection box is less than a sixth threshold;

[0174] Thirdly, the average level of infrared radiation intensity of the image area represented by the first mask information corresponding to the detection frame is less than a seventh threshold.

[0175] The fifth, sixth, or seventh thresholds can be set based on experience and actual conditions. Specifically, when identifying animals in a forest scene, most falsely detected interference objects are tree trunks, leaves, stones, etc. Tree trunks are long and slender, so the area ratio between the area represented by the first mask information and the area represented by the detection frame is small, and the ratio of the short side to the long side of the detection frame is also small. Detection frames and the corresponding first mask information with an area ratio less than the fifth threshold can be deleted, and detection frames and the corresponding first mask information with a short side to long side ratio less than the sixth threshold can also be deleted. For example, the fifth threshold can be 0.3, and the sixth threshold can be 0.2.

[0176] In addition, the surface temperature of the falsely detected interference object is generally lower than that of the animal, and the average level of infrared radiation intensity of the image area represented by the corresponding falsely detected first mask information will also be lower. Therefore, if the average level of infrared radiation intensity of the image area belonging to the target of interest represented by the first mask information corresponding to a detection frame is less than the seventh threshold, the detection frame and the corresponding first mask information can be deleted.

[0177] It is understood that when the image frame to be processed is a thermal image, the average level of infrared radiation intensity of the image area represented by the first mask information in the image frame to be processed may be the average value of the pixel values ​​within that image area. When the image frame to be processed is a visible light image, the infrared radiation intensity of the image area represented by the first mask information in the image frame to be processed may be determined in conjunction with a simultaneously acquired thermal image. For example, the thermal image and the image frame to be processed may be acquired for the same scene. The corresponding region in the thermal image may then be determined based on the image area represented by the first mask information, and the average value of the pixel values ​​within that region may be calculated to obtain the infrared radiation intensity of the image area represented by the first mask information.

[0178] S303, determining, from the current cache set, cache frames that are different from the objects of interest represented by each detection frame to be used as cache frames to be processed;

[0179] The cache frames in the current cache set include: detection frames representing different objects of interest identified in a first number of historical image frames adjacent to the image frame to be processed.

[0180] S304, performing image segmentation on the image region corresponding to each buffer frame to be processed in the image frame to be processed, and obtaining second mask information representing the target image region occupied by the object of interest in the image region;

[0181] S305 : In the image frame to be processed, highlight the target image region represented by the obtained first mask information and the second mask information.

[0182] The above steps S303-S305 are similar to steps S102-S104 and will not be repeated here.

[0183] In one embodiment of the present application, Figure 4 As shown, before performing target recognition on the image frame to be processed in the video to be processed and obtaining the detection frame to be used of the target of interest in the image frame to be processed and the corresponding first mask information, the method further includes:

[0184] S401, determining status information corresponding to the image frame to be processed;

[0185] The status information corresponding to an image frame indicates whether the image acquisition device was in motion when the image frame was acquired. The image acquisition device may be provided with a gyroscope, and the status information indicating whether the image acquisition device was in motion when acquiring the image frame to be processed may be determined based on gyroscope data collected by the gyroscope when acquiring the image frame to be processed.

[0186] In one implementation, the determining of the status information corresponding to the image frame to be processed includes:

[0187] Step C1, acquiring multiple angular velocities collected by the gyroscope of the image acquisition device for each acquisition direction during the period from the acquisition of the previous image frame to the acquisition of the image frame to be processed;

[0188] Step C2: for each acquisition direction, calculating the sum of the angular velocities acquired by the gyroscope for the acquisition direction, and obtaining first motion information for each acquisition direction corresponding to the image frame to be processed;

[0189] Because the gyroscope's acquisition cycle is shorter than that of the image frame, the gyroscope can acquire multiple sets of gyroscope data within each image frame's acquisition cycle. Each set of gyroscope data contains angular velocities in three acquisition directions: pitch (forward and backward tilt), yaw (left and right swing), and roll (left and right tilt). In specific implementations, the absolute value of the sum of the angular velocities in each acquisition direction can be calculated for these multiple sets of gyroscope data. This value reflects the integral of the velocity in that acquisition direction within a single frame period and also reflects the distance the image acquisition device travels per unit time. Since the purpose is only to determine whether the image acquisition device is stable, it is not sensitive to direction and the absolute value of the sum of the angular velocities can be directly taken.

[0190] Step C3: for each acquisition direction, calculating the range of the angular velocity acquired by the gyroscope for the acquisition direction, and calculating the sum of the ranges of the acquired ranges for each acquisition direction to obtain second motion information corresponding to the image frame to be processed;

[0191] The sum of the ranges in each acquisition direction can reflect the overall motion acceleration of the image acquisition device. By determining the first motion information and second motion information corresponding to the image frame to be processed, the motion of the image acquisition device can be more comprehensively reflected, thereby improving the accuracy of the determined motion state.

[0192] Step C4 : determining the state information corresponding to the image frame to be processed based on the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed.

[0193] In one implementation, when the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed are both greater than a preset threshold, it can be determined that the image acquisition device is in motion when acquiring the image frame to be processed.

[0194] In another implementation, the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed can also be input into a pre-trained decision model to obtain the status information corresponding to the image frame to be processed output by the decision model.

[0195] The decision model may be a decision tree model, such as a random forest model, etc. The decision model may be a model trained based on the first motion information of each acquisition direction corresponding to the sample image acquired by the sample image acquisition device and the second motion information corresponding to the sample image, as well as the actual state of the sample image acquisition device when acquiring the sample image.

[0196] The sample image acquisition device can be an image acquisition device that acquires image frames to be processed, or it can be an image acquisition device of other similar specifications. Specifically, the image frames acquired when the sample image acquisition device is stationary can be used as positive sample images, in which case the actual state of the sample image acquisition device is a stable state; and the image frames acquired when the image acquisition device is in motion can be used as negative sample images, in which case the actual state of the sample image acquisition device is an unstable state; and then the first motion information and the corresponding second motion information of each acquisition direction corresponding to the positive sample image are obtained; and the first motion information and the corresponding second motion information of each acquisition direction corresponding to the negative sample image are obtained, respectively, as positive sample data and negative sample data, and then based on the positive sample data and the negative sample data, the decision model to be trained is trained to obtain a trained decision model.

[0197] S402, based on the state information corresponding to the image frame to be processed, determining whether the image acquisition device is in a stable state when acquiring the image frame to be processed;

[0198] In one implementation, when the state information corresponding to the image frame to be processed indicates a motion state, it may be determined whether the image acquisition device is in a stable state when acquiring the image frame to be processed.

[0199] It is understandable that the status information corresponding to an image frame is often accidental. If the image acquisition device is in a stable state based only on the status information corresponding to an image frame, it may cause the function of highlighting the image frame target to be frequently turned on and off, which may also cause the video screen to flicker.

[0200] Therefore, in another implementation, a smoothing process can be used to determine whether the image acquisition device is in a stable state. Specifically, based on the state information corresponding to the image frame to be processed, determining whether the image acquisition device is in a stable state when the image frame to be processed is captured includes one of the following situations:

[0201] Case 1: When the image acquisition device was in a stable state when acquiring the image frame preceding the image frame to be processed, if the state information corresponding to the image frame to be processed and a second number of adjacent historical image frames all indicate a motion state, it is determined that the image acquisition device was in an unstable state when acquiring the image frame to be processed;

[0202] That is, when the state information corresponding to a plurality of consecutive image frames all indicates a motion state, the state of the image acquisition device switches from a stable state to an unstable state.

[0203] Otherwise, if the state information corresponding to any one of the image frame to be processed and the adjacent second number of historical image frames indicates a non-motion state, it is determined that the image acquisition device was in a stable state when acquiring the current image frame to be processed, that is, there is no need to switch states at this time;

[0204] Case 2: When the image acquisition device was in an unstable state when acquiring the image frame preceding the image frame to be processed, if the state information corresponding to the image frame to be processed and a second number of adjacent historical image frames all indicate a non-motion state, then it is determined that the image acquisition device was in a stable state when acquiring the image frame to be processed;

[0205] That is, when the state information corresponding to a plurality of consecutive image frames all indicates a non-motion state, the state of the image acquisition device switches from an unstable state to a stable state.

[0206] Otherwise, if the state information corresponding to any image frame among the image frame to be processed and the adjacent second number of historical image frames indicates a motion state, it is determined that the image acquisition device is in an unstable state when acquiring the current image frame to be processed, that is, there is no need to switch the state at this time.

[0207] By using the above-mentioned smoothing method, it is possible to avoid frequently starting and stopping the target highlighting function, thereby further reducing the flickering phenomenon of the video image and improving the stability of the video image.

[0208] For example, if the second number is 10, when it is necessary to perform target highlighting on image frames in a video to be processed, detection can be started from the first image frame in the video to be processed, and until the state information corresponding to 10 consecutive image frames is detected to indicate a non-motion state, target highlighting can be started from the last image frame of the 10 image frames. Thereafter, until the state information corresponding to 10 consecutive image frames is detected to indicate a motion state, target highlighting can be stopped for the last image frame of the 10 image frames, and target highlighting can be started again when the state information corresponding to 10 consecutive image frames is detected to indicate a non-motion state, and so on.

[0209] If it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed, S403 is executed. If it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed, the process of processing the image to be processed is terminated.

[0210] S403 : performing target recognition on the image frame to be processed in the video to be processed, and obtaining a detection frame to be used of the target of interest in the image frame to be processed and corresponding first mask information.

[0211] After S403, the above steps S102-S104 may be continued to be executed.

[0212] If it is determined that the image acquisition device is in an unstable state when acquiring the image frame to be processed, the process of performing target image processing on the image frame to be processed may be terminated.

[0213] It is understandable that for each image frame in the video to be processed, each image frame can be taken as an image frame to be processed in turn, and judged according to the above steps S401-S402. When it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed, the process of highlighting the target of the image frame to be processed is executed. When it is in an unstable state, the target of the image frame to be processed is not highlighted.

[0214] For non-fixed image acquisition devices, when the image acquisition device is in a state of relatively violent shaking, the video screen will be shaken, causing the target of interest to continuously move in the video screen, increasing the difficulty of target recognition. Target enhancement of such a video will also lead to instability of the video screen. Therefore, this embodiment determines whether the image acquisition device is in a stable state when the image frame to be processed is acquired based on the state information corresponding to the image frame to be processed; and then, when it is determined that the image acquisition device is in a stable state when the image frame to be processed is acquired, the above-mentioned process of target recognition and target highlighting is executed for the image frame to be processed in the video to be processed. When the image acquisition device is in an unstable state when the image frame to be processed is acquired, target recognition and target highlighting may not be performed. Therefore, the stability of the video screen can be further improved by this embodiment.

[0215] For example, the overall process of the embodiment of the present application can be as follows: Figure 5 As shown:

[0216] In this embodiment, the input data is: gyroscope data 10 and 8-bit thermal imaging data 20. In this case, the 8-bit thermal imaging data 20 is the image frame to be processed. First, S501 is executed to determine whether the image acquisition device is in a stable state when the image frame to be processed is acquired. If stable, S502 is executed; if not, S503 is executed.

[0217] S502, function is turned on; S503, function is turned off; that is, the subsequent process is no longer executed for the image frame to be processed.

[0218] S504, segmentation model processing to obtain a mask 30 and a detection frame 40; that is, using the target segmentation model to perform target recognition on the image frame to be processed, to obtain a detection frame to be used for the target of interest in the image frame to be processed and corresponding first mask information;

[0219] S505, false detection filtering, obtaining a filtered detection frame 50 and a filtered mask 60; that is, deleting the detection frames that meet the preset deletion conditions from the obtained initial detection frame, and using the remaining detection frames as the detection frames to be used, to obtain the detection frame to be used of the target of interest in the image frame to be processed and the corresponding first mask information.

[0220] S506, target matching; that is, for each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a candidate cache frame;

[0221] S507, determining whether there is any missed detection; that is, for each candidate cache frame, determining whether there is mask information in the first mask information that matches the candidate cache frame; if not, it indicates that the image region corresponding to the candidate cache frame in the image frame to be processed has missed detection, and the candidate cache frame is used as the cache frame to be processed, and executing S508; if it does, it indicates that the image region corresponding to the cache frame to be processed in the image frame to be processed has not missed detection, and no further processing is required for the image region;

[0222] S508: Threshold segmentation is performed to obtain a final mask 70 and a final detection frame 80. Specifically, if the image region corresponding to the buffer frame to be processed has missed detections, image segmentation is performed on that image region to obtain second mask information representing the target image region occupied by the object of interest within that image region. The final mask 70 includes the obtained first and second mask information. The final detection frame 80 includes the obtained detection frame to be used and the buffer frame to be processed corresponding to the missed detection region.

[0223] Afterwards, the target image region represented by the obtained first mask information and the second mask information may be highlighted in the image frame to be processed, and the process ends.

[0224] In this embodiment, deep learning-based object segmentation is combined with an adaptive highlighting scheme to design a corresponding false detection filtering scheme and missed detection compensation scheme. This not only makes the detection of objects of interest more stable, but also significantly improves segmentation accuracy and robustness compared to traditional threshold segmentation schemes. Secondly, by using gyroscope data to determine the state of the image acquisition device, the problem of unstable detection caused by image jitter is avoided.

[0225] The embodiment of the present application also provides a target highlighting device, such as Figure 6 As shown, the device includes:

[0226] The target recognition module 601 is configured to perform target recognition on an image frame to be processed in a video to be processed, and obtain a detection frame to be used and corresponding first mask information of an object of interest in the image frame to be processed; wherein the first mask information corresponding to the detection frame of the object of interest represents: a target image area occupied by the object of interest within the detection frame;

[0227] The to-be-processed cache frame determination module 602 is configured to determine, from the current cache set, cache frames that are different from the objects of interest represented by each detection frame to be used, as the to-be-processed cache frames; wherein the cache frames in the current cache set include detection frames representing different objects of interest identified in a first number of historical image frames adjacent to the to-be-processed image frame;

[0228] An image segmentation module 603 is configured to perform image segmentation on an image region corresponding to each buffer frame to be processed in the image frame to be processed, and obtain second mask information representing a target image region occupied by an object of interest in the image region;

[0229] The highlighting module 604 is configured to highlight the target image region represented by the obtained first mask information and the second mask information in the image frame to be processed.

[0230] In one embodiment, the module 602 for determining a cache frame to be processed includes:

[0231] a candidate cache frame determination submodule, configured to determine, for each cache frame in the current cache set, a cache frame as a candidate cache frame if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio;

[0232] a buffer frame to be processed determination submodule, configured to, for each candidate buffer frame, determine the candidate buffer frame as a buffer frame to be processed if no mask information matching the candidate buffer frame exists in the first mask information; wherein the matching mask information and the category of the target of interest represented by the buffer frame are consistent, and a portion of the target image region represented by the mask information within the buffer frame is greater than a first threshold;

[0233] In one embodiment, the buffer frame to be processed determination module 602 is specifically configured to:

[0234] For each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a cache frame to be processed;

[0235] In one embodiment, the apparatus further comprises:

[0236] The cache frame deletion module is used to delete each cache frame in the current cache set from the current cache set if the cache frame meets any of the following conditions:

[0237] Among the obtained detection frames to be used, there is a detection frame to be used whose intersection-over-union ratio with the cache frame is greater than a preset intersection-over-union ratio; the cache frame and the detection frames identified in the first number of image frames to be compared represent different objects of interest; wherein the image frames to be compared include the image frame to be processed and adjacent historical image frames;

[0238] The cache frame adding module is used to add the obtained detection frame to be used as a cache frame to the current cache set.

[0239] In one embodiment, the intersection-over-union (IoU) ratio between the detection frame and the cache frame is greater than a preset IoU ratio and satisfies at least one of the following conditions:

[0240] The intersection-and-union ratio of the image areas represented by the detection frame and the cache frame is greater than the second threshold; the intersection-and-union ratio of the target image areas represented by the first mask information corresponding to the detection frame and the cache frame is greater than the third threshold; the intersection-and-union ratio of the extended image areas corresponding to the detection frame and the cache frame is greater than the fourth threshold.

[0241] In one embodiment, the image segmentation module 603 is specifically configured to:

[0242] For the image area corresponding to each cache frame to be processed in the image frame to be processed, the image area is segmented using the maximum entropy threshold segmentation algorithm based on the gradient of each pixel point in the image area to obtain second mask information representing the target image area occupied by the target of interest in the image area.

[0243] In one embodiment, the target identification module 601 includes:

[0244] The target recognition submodule is used to perform target recognition on the image frame to be processed in the video to be processed, and obtain an initial detection frame of the target of interest in the image frame to be processed and corresponding first mask information;

[0245] The screening submodule is used to delete the detection frames that meet the preset deletion conditions from the obtained initial detection frames, and use the remaining detection frames as the detection frames to be used, to obtain the detection frames to be used of the target of interest in the image frame to be processed and the corresponding first mask information.

[0246] In one embodiment, the target of interest is an animal, and the preset deletion condition includes at least one of the following:

[0247] The ratio of the area represented by the first mask information corresponding to the detection frame to the area represented by the detection frame is less than a fifth threshold;

[0248] The ratio of the short side to the long side of the detection frame is less than a sixth threshold;

[0249] The average level of infrared radiation intensity of the image area represented by the first mask information corresponding to the detection frame is less than a seventh threshold.

[0250] In one embodiment, the apparatus further comprises:

[0251] a state information determination module configured to determine state information corresponding to the image frame to be processed before the target recognition module performs target recognition on the image frame to be processed in the video to be processed and obtains a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed; wherein the state information corresponding to an image frame indicates whether the image acquisition device was in motion when the image frame was acquired;

[0252] a stable state judgment module, configured to determine whether the image acquisition device is in a stable state when acquiring the image frame to be processed based on state information corresponding to the image frame to be processed;

[0253] The target recognition module 601 is specifically used to perform target recognition on the image frame to be processed in the video to be processed when it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed, and obtain a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed.

[0254] In one embodiment, the stable state judgment module is specifically configured to:

[0255] In a case where the image acquisition device was in a stable state when acquiring the previous image frame of the image frame to be processed, if the state information corresponding to the image frame to be processed and a second number of adjacent historical image frames all indicate a motion state, then it is determined that the image acquisition device was in an unstable state when acquiring the image frame to be processed; if the state information corresponding to any one of the image frame to be processed and the second number of adjacent historical image frames indicates a non-motion state, then it is determined that the image acquisition device was in a stable state when acquiring the image frame to be processed;

[0256] In the case that the image acquisition device is in an unstable state when acquiring the previous image frame of the image frame to be processed, if the status information corresponding to the image frame to be processed and the second number of adjacent historical image frames all indicate a non-motion state, it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed; if the status information corresponding to any image frame among the image frame to be processed and the second number of adjacent historical image frames indicates a motion state, it is determined that the image acquisition device is in an unstable state when acquiring the image frame to be processed.

[0257] In one embodiment, the status information determination module includes:

[0258] an angular velocity acquisition submodule, configured to acquire a plurality of angular velocities acquired by the gyroscope of the image acquisition device for each acquisition direction during a period from the acquisition of the previous image frame to the acquisition of the image frame to be processed;

[0259] a first calculation submodule, configured to calculate, for each acquisition direction, the sum of the angular velocities acquired by the gyroscope in the acquisition direction, and obtain first motion information in each acquisition direction corresponding to the image frame to be processed;

[0260] a second calculation submodule, configured to calculate, for each acquisition direction, a range of angular velocities acquired by the gyroscope in the acquisition direction, and calculate a sum of the obtained ranges for each acquisition direction to obtain second motion information corresponding to the image frame to be processed;

[0261] The state information determination submodule is configured to determine the state information corresponding to the image frame to be processed based on the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed.

[0262] In one embodiment, the state information determination submodule is specifically configured to:

[0263] The first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed are input into a pre-trained decision model to obtain the state information corresponding to the image frame to be processed output by the decision model; wherein the decision model is: a model trained based on the first motion information of each acquisition direction corresponding to the sample image acquired by the sample image acquisition device and the second motion information corresponding to the sample image, and the actual state of the sample image acquisition device when acquiring the sample image.

[0264] The present application also provides an electronic device, such as Figure 7 Shown, including:

[0265] Memory 701, used for storing computer programs;

[0266] The processor 702 is configured to implement any of the steps of the above-mentioned target highlighting method when executing the program stored in the memory 701 .

[0267] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 702, the communication interface, and the memory 701 communicate with each other via the communication bus.

[0268] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0269] The communication interface is used for communication between the above electronic device and other devices.

[0270] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0271] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0272] In another embodiment provided by the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps of any of the above-mentioned target highlighting methods are implemented.

[0273] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any target highlighting method described in any of the above embodiments.

[0274] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state drive (SSD).

[0275] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0276] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from other embodiments. In particular, the device, electronic device, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0277] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A target highlighting method, characterized in that: The method comprises: Performing target recognition on an image frame to be processed in a video to be processed, obtaining a detection frame to be used and corresponding first mask information of an object of interest in the image frame to be processed; wherein the first mask information corresponding to the detection frame of the object of interest represents: a target image area occupied by the object of interest within the detection frame; Determining, from a current cache set, cache frames that are different from the objects of interest represented by each detection frame to be used as cache frames to be processed; wherein the cache frames in the current cache set include: detection frames representing different objects of interest identified in a first number of historical image frames adjacent to the image frame to be processed; Performing image segmentation on an image region corresponding to each buffer frame to be processed in the image frame to be processed, and obtaining second mask information representing a target image region occupied by an object of interest within the image region; In the image frame to be processed, the target image region represented by the obtained first mask information and the second mask information is highlighted.

2. The method according to claim 1, characterized in that The step of determining, from the current cache set, cache frames that are different from the target of interest represented by each detection frame to be used as cache frames to be processed includes: For each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a candidate cache frame; For each candidate cache frame, if there is no mask information matching the candidate cache frame in the first mask information, the candidate cache frame is determined as the cache frame to be processed; wherein the matching mask information and the category of the target of interest represented by the cache frame are consistent, and the portion of the target image area represented by the mask information within the cache frame is greater than a first threshold.

3. The method according to claim 1, characterized in that The step of determining, from the current cache set, cache frames that are different from the target of interest represented by each detection frame to be used as cache frames to be processed includes: For each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a cache frame to be processed.

4. The method according to claim 1, wherein The method further comprises: For each cache frame in the current cache set, if the cache frame meets any of the following conditions, the cache frame will be deleted from the current cache set: Among the obtained detection frames to be used, there is a detection frame to be used whose intersection-over-union ratio with the cache frame is greater than a preset intersection-over-union ratio; The cache frame and the detection frames identified in the first number of image frames to be compared represent different objects of interest; wherein the image frames to be compared include the image frame to be processed and adjacent historical image frames; The obtained detection frame to be used is added as a cache frame to the current cache set.

5. The method according to any one of claims 2 to 4, characterized in that: The intersection-over-union (IoU) ratio between the detection frame and the cache frame is greater than the preset IoU ratio and satisfies at least one of the following conditions: The intersection-over-union ratio of the image area represented by the detection frame and the cache frame is greater than a second threshold; The intersection-over-union ratio of the target image area represented by the first mask information corresponding to the detection frame and the cache frame is greater than a third threshold; An intersection-over-union ratio of the extended image areas corresponding to the detection frame and the cache frame is greater than a fourth threshold.

6. The method according to claim 1, wherein Performing image segmentation on the image region corresponding to each buffer frame to be processed in the image frame to be processed to obtain second mask information representing the target image region occupied by the target of interest in the image region, including: For the image area corresponding to each cache frame to be processed in the image frame to be processed, the image area is segmented using the maximum entropy threshold segmentation algorithm based on the gradient of each pixel point in the image area to obtain second mask information representing the target image area occupied by the target of interest in the image area.

7. The method according to claim 1, characterized in that The method of performing target recognition on the image frame to be processed in the video to be processed and obtaining a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed includes: Performing target recognition on the image frame to be processed in the video to be processed, and obtaining an initial detection frame of the target of interest in the image frame to be processed and corresponding first mask information; Detection frames that meet a preset deletion condition are deleted from the obtained initial detection frames, and the remaining detection frames are used as detection frames to be used, thereby obtaining the detection frame to be used of the target of interest in the image frame to be processed and the corresponding first mask information.

8. The method according to claim 7, characterized in that The target of interest is an animal, and the preset deletion condition includes at least one of the following: The ratio of the area represented by the first mask information corresponding to the detection frame to the area represented by the detection frame is less than a fifth threshold; The ratio of the short side to the long side of the detection frame is less than a sixth threshold; The average level of infrared radiation intensity of the image area represented by the first mask information corresponding to the detection frame is less than a seventh threshold.

9. The method according to claim 1, characterized in that Before performing target recognition on the image frame to be processed in the video to be processed and obtaining a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed, the method further includes: Determining state information corresponding to the image frame to be processed; wherein the state information corresponding to an image frame indicates whether the image acquisition device is in motion when acquiring the image frame; determining, based on the state information corresponding to the image frame to be processed, whether the image acquisition device is in a stable state when acquiring the image frame to be processed; The method of performing target recognition on the image frame to be processed in the video to be processed and obtaining a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed includes: When it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed, target recognition is performed on the image frame to be processed in the video to be processed to obtain a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed.

10. The method according to claim 9, characterized in that The determining, based on the state information corresponding to the image frame to be processed, whether the image acquisition device is in a stable state when acquiring the image frame to be processed includes: In a case where the image acquisition device was in a stable state when acquiring the previous image frame of the image frame to be processed, if the state information corresponding to the image frame to be processed and a second number of adjacent historical image frames all indicate a motion state, then it is determined that the image acquisition device was in an unstable state when acquiring the image frame to be processed; if the state information corresponding to any one of the image frame to be processed and the second number of adjacent historical image frames indicates a non-motion state, then it is determined that the image acquisition device was in a stable state when acquiring the image frame to be processed; In the case that the image acquisition device is in an unstable state when acquiring the previous image frame of the image frame to be processed, if the status information corresponding to the image frame to be processed and the second number of adjacent historical image frames all indicate a non-motion state, it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed; if the status information corresponding to any image frame among the image frame to be processed and the second number of adjacent historical image frames indicates a motion state, it is determined that the image acquisition device is in an unstable state when acquiring the image frame to be processed.

11. The method according to claim 9, characterized in that The determining of the status information corresponding to the image frame to be processed includes: Acquiring multiple angular velocities collected by the gyroscope of the image acquisition device for each acquisition direction during a period from the acquisition of the previous image frame to the acquisition of the image frame to be processed; For each acquisition direction, calculating the sum of the angular velocities acquired by the gyroscope for the acquisition direction, and obtaining first motion information of each acquisition direction corresponding to the image frame to be processed; For each acquisition direction, calculating the range of the angular velocity acquired by the gyroscope for the acquisition direction, and calculating the sum of the obtained ranges of each acquisition direction to obtain second motion information corresponding to the image frame to be processed; Based on the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed, the state information corresponding to the image frame to be processed is determined.

12. The method according to claim 11, characterized in that The determining, based on the first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed, the state information corresponding to the image frame to be processed includes: The first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed are input into a pre-trained decision model to obtain the state information corresponding to the image frame to be processed output by the decision model; wherein the decision model is: a model trained based on the first motion information of each acquisition direction corresponding to the sample image acquired by the sample image acquisition device and the second motion information corresponding to the sample image, and the actual state of the sample image acquisition device when acquiring the sample image.

13. A target highlighting device, characterized in that: The device comprises: a target recognition module configured to perform target recognition on an image frame to be processed in a video to be processed, and obtain a detection frame to be used and corresponding first mask information of an object of interest in the image frame to be processed; wherein the first mask information corresponding to the detection frame of the object of interest represents: a target image area occupied by the object of interest within the detection frame; a buffer frame determination module for processing, configured to determine, from a current buffer set, buffer frames that are different from the objects of interest represented by each detection frame to be used, as buffer frames to be processed; wherein the buffer frames in the current buffer set include detection frames representing different objects of interest identified in a first number of historical image frames adjacent to the image frame to be processed; An image segmentation module is used to perform image segmentation on an image region corresponding to each buffer frame to be processed in the image frame to be processed, and obtain second mask information representing a target image region occupied by an object of interest in the image region; The highlighting module is used to highlight the target image area represented by the obtained first mask information and the second mask information in the image frame to be processed.

14. The device according to claim 13, characterized in that The module for determining a cache frame to be processed includes: a candidate cache frame determination submodule, configured to determine, for each cache frame in the current cache set, a cache frame as a candidate cache frame if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio; a buffer frame to be processed determination submodule, configured to, for each candidate buffer frame, determine the candidate buffer frame as a buffer frame to be processed if no mask information matching the candidate buffer frame exists in the first mask information; wherein the matching mask information and the category of the target of interest represented by the buffer frame are consistent, and a portion of the target image region represented by the mask information within the buffer frame is greater than a first threshold; and / or, The module for determining the cache frame to be processed is specifically configured to: For each cache frame in the current cache set, if the intersection-and-union ratio between the cache frame and each detection frame to be used is not greater than a preset intersection-and-union ratio, the cache frame is determined as a cache frame to be processed; and / or, The device further comprises: The cache frame deletion module is configured to delete each cache frame in the current cache set from the current cache set if the cache frame satisfies any of the following conditions: Among the obtained detection frames to be used, there is a detection frame to be used whose intersection-over-union ratio with the cache frame is greater than a preset intersection-over-union ratio; The cache frame and the detection frames identified in the first number of image frames to be compared represent different objects of interest; wherein the image frames to be compared include the image frame to be processed and adjacent historical image frames; A cache frame adding module is used to add the obtained detection frame to be used as a cache frame to the current cache set; and / or, The intersection-over-union (IoU) ratio between the detection frame and the cache frame is greater than the preset IoU ratio and satisfies at least one of the following conditions: The intersection-over-union ratio of the image area represented by the detection frame and the cache frame is greater than a second threshold; The intersection-over-union ratio of the target image area represented by the first mask information corresponding to the detection frame and the cache frame is greater than a third threshold; The intersection-over-union ratio of the extended image areas corresponding to the detection frame and the cache frame is greater than a fourth threshold; and / or, The image segmentation module is specifically used to: For an image region corresponding to each buffer frame to be processed in the image frame to be processed, a maximum entropy threshold segmentation algorithm is used to segment the image region based on the gradient of each pixel point in the image region to obtain second mask information representing a target image region occupied by an object of interest in the image region; and / or, The target recognition module includes: The target recognition submodule is used to perform target recognition on the image frame to be processed in the video to be processed, and obtain an initial detection frame of the target of interest in the image frame to be processed and corresponding first mask information; a screening submodule, configured to delete the detection frames that meet a preset deletion condition from the obtained initial detection frames, and use the remaining detection frames as the detection frames to be used, thereby obtaining the detection frames to be used of the target of interest in the image frame to be processed and the corresponding first mask information; and / or, The target of interest is an animal, and the preset deletion condition includes at least one of the following: The ratio of the area represented by the first mask information corresponding to the detection frame to the area represented by the detection frame is less than a fifth threshold; The ratio of the short side to the long side of the detection frame is less than a sixth threshold; The average level of infrared radiation intensity of the image area represented by the first mask information corresponding to the detection frame is less than a seventh threshold; and / or, The device further comprises: a state information determination module configured to determine state information corresponding to the image frame to be processed before the target recognition module performs target recognition on the image frame to be processed in the video to be processed and obtains a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed; wherein the state information corresponding to an image frame indicates whether the image acquisition device was in motion when the image frame was acquired; a stable state judgment module, configured to determine whether the image acquisition device is in a stable state when acquiring the image frame to be processed based on state information corresponding to the image frame to be processed; The target recognition module is specifically configured to, when it is determined that the image acquisition device is in a stable state when acquiring the image frame to be processed, perform target recognition on the image frame to be processed in the video to be processed, and obtain a detection frame to be used and corresponding first mask information of the target of interest in the image frame to be processed; and / or, The stable state judgment module is specifically used to: In a case where the image acquisition device was in a stable state when acquiring the previous image frame of the image frame to be processed, if the state information corresponding to the image frame to be processed and a second number of adjacent historical image frames all indicate a motion state, then it is determined that the image acquisition device was in an unstable state when acquiring the image frame to be processed; if the state information corresponding to any one of the image frame to be processed and the second number of adjacent historical image frames indicates a non-motion state, then it is determined that the image acquisition device was in a stable state when acquiring the image frame to be processed; In a case where the image acquisition device was in an unstable state when acquiring the previous image frame of the image frame to be processed, if the state information corresponding to the image frame to be processed and a second number of adjacent historical image frames all indicate a non-motion state, then it is determined that the image acquisition device was in a stable state when acquiring the image frame to be processed; if the state information corresponding to any one of the image frame to be processed and the second number of adjacent historical image frames indicates a motion state, then it is determined that the image acquisition device was in an unstable state when acquiring the image frame to be processed; and / or, The status information determination module includes: an angular velocity acquisition submodule, configured to acquire a plurality of angular velocities acquired by the gyroscope of the image acquisition device for each acquisition direction during a period from the acquisition of the previous image frame to the acquisition of the image frame to be processed; a first calculation submodule, configured to calculate, for each acquisition direction, the sum of the angular velocities acquired by the gyroscope in the acquisition direction, and obtain first motion information in each acquisition direction corresponding to the image frame to be processed; a second calculation submodule, configured to calculate, for each acquisition direction, a range of angular velocities acquired by the gyroscope in the acquisition direction, and calculate a sum of the obtained ranges for each acquisition direction to obtain second motion information corresponding to the image frame to be processed; a state information determination submodule, configured to determine state information corresponding to the image frame to be processed based on first motion information of each acquisition direction corresponding to the image frame to be processed and second motion information corresponding to the image frame to be processed; and / or, The state information determination submodule is specifically used to: The first motion information of each acquisition direction corresponding to the image frame to be processed and the second motion information corresponding to the image frame to be processed are input into a pre-trained decision model to obtain the state information corresponding to the image frame to be processed output by the decision model; wherein the decision model is: a model trained based on the first motion information of each acquisition direction corresponding to the sample image acquired by the sample image acquisition device and the second motion information corresponding to the sample image, and the actual state of the sample image acquisition device when acquiring the sample image.

15. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 12 when executing a program stored in a memory.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

17. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 12.

Images