Turbulence state identification method and device, target tracking method and device

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to a method and device for identifying a jolting state, and a method and device for tracking a target. The method for identifying the jolting state comprises: collecting a current frame image and a historical frame image containing a tracking target; estimating an estimated bounding box of each tracking target in the current frame image based on the historical frame image; obtaining an actual bounding box of each tracking target from the current frame image; obtaining a matching number of the estimated bounding box and the actual bounding box; and determining that a camera is in a jolting state if the matching number is zero. The method can reduce the probability of target tracking failure caused by not taking a correct tracking method in time due to inaccurate jolting state determination.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a jolt state recognition method, a jolt state recognition device, a target tracking method and a target tracking device. BACKGROUND

[0002] Multiple Object Tracking (MOT) is now widely used in automatic driving, intelligent transportation and other fields. The main purpose of MOT is to give a time-ordered image sequence, obtain the target objects in each image through target detection technology, and then correspond the same object in different images to obtain the time information and spatial information of the target.

[0003] In the target tracking process, the core algorithm is the target matching algorithm, which matches the same target between the previous and the next two frames. The matching strategy is generally to match and track the target according to the intersection-over-union between the areas of the motion target detection boxes in adjacent frames. However, in actual application, the camera often jolts due to environmental reasons, causing the target boxes between the two frames to be too different, resulting in a too small or even no intersection-over-union, and causing the target tracking to fail. Although in the prior art, different tracking methods are used when the camera jolts to avoid the problem of tracking failure, the accuracy of the judgment of whether the camera jolts in the prior art is insufficient, and there is a case where the camera actually (does not / does) jolt, but the algorithm considers that it (does not / does) jolt, resulting in misjudgment and causing the subsequent process to adopt a non-correct tracking method, and thus leading to target tracking failure. Therefore, the state of the camera needs to be accurately judged before target tracking. SUMMARY

[0004] One of the purposes of the present application is to provide a jolt state recognition method to reduce the probability of target tracking failure caused by inaccurate jolt state judgment and failure to adopt a correct tracking method. The second purpose is to provide a jolt state recognition device. The third purpose is to provide a target tracking method to improve the stability and robustness of target tracking. The fourth purpose is to provide a target tracking device.

[0005] To achieve the above purposes, the first aspect of the present application provides a jolt state recognition method applicable to a camera in a moving state, which comprises: collecting a current frame image and a historical frame image containing tracking targets; estimating an estimated bounding box of each tracking target in the current frame image based on the historical frame image; obtaining an actual bounding box of each tracking target from the current frame image; obtaining the number of tracking targets whose estimated bounding box matches the actual bounding box, and obtaining a matching number; and determining that the camera is in a jolt state if the matching number is zero.

[0006] Based on the first aspect, in some embodiments of the present application, the method for determining whether the estimated bounding box matches the actual bounding box comprises: calculating an intersection-over-union of areas of the estimated bounding box and the actual bounding box in the current frame image; determining whether the intersection-over-union is greater than a threshold value; if yes, the estimated bounding box of the tracking target matches the actual bounding box; otherwise, the estimated bounding box of the tracking target does not match the actual bounding box.

[0007] Based on the first aspect, in some embodiments of the present application, the method for identifying the jolting state further comprises: obtaining a total number of the actual bounding boxes; if the matching number is not zero but less than the total number, a preset verification method is used to further verify whether the camera is in the jolting state, specifically comprising:

[0008] selecting stable tracking targets with a number of historical matching successes greater than a preset number from the tracking targets; calculating an average value of pixel heights of the estimated bounding boxes of the stable tracking targets in the current frame image to obtain a first average value; calculating an average value of pixel heights of the actual bounding boxes of the stable tracking targets in the current frame image to obtain a second average value; calculating a difference value between the first average value and the second average value; determining whether the difference value is greater than a preset value; if yes, the camera is in the jolting state; otherwise, the camera is in the non-jolting state; in addition, if the matching number is equal to the total number, the camera is in the non-jolting state.

[0009] The second aspect of the present application provides a jolting state identification device suitable for a camera in a moving state, which comprises: a collection module for collecting a current frame image and historical frame images containing tracking targets; an estimation module for estimating an estimated bounding box of each tracking target in the current frame image based on the historical frame images; a first acquisition module for acquiring an actual bounding box of each tracking target from the current frame image; a second acquisition module for acquiring a number of tracking targets whose estimated bounding box matches the actual bounding box, to obtain a matching number; and a first output module for outputting a recognition result that the camera is in a jolting state when the matching number is zero.

[0010] Based on the second aspect, in some embodiments of the present application, the second acquisition module comprises: a judgment unit group for determining whether the estimated bounding box matches the actual bounding box; the judgment unit group comprises: a first calculation unit for calculating an intersection-over-union of areas of the estimated bounding box and the actual bounding box in the current frame image; a first judgment unit for determining whether the intersection-over-union is greater than a threshold value; a first output unit for outputting a determination conclusion that the estimated bounding box of the tracking target matches the actual bounding box when the intersection-over-union is greater than the threshold value; and a second output unit for outputting a determination conclusion that the estimated bounding box of the tracking target does not match the actual bounding box when the intersection-over-union is not greater than the threshold value.

[0011] In some embodiments of the second aspect, the jolt state identification device further comprises a third acquisition module configured to acquire a total number of actual bounding boxes; a verification module configured to further verify whether the camera is in a jolt state using a preset verification method when the matching number is not zero but less than the total number; the verification module comprises a selection unit configured to select stable tracking targets with a number of historical matching successes greater than a preset number from the tracking targets; a second calculation unit configured to calculate an average value of pixel heights of the estimated bounding boxes of the stable tracking targets in the current frame image to obtain a first average value; a third calculation unit configured to calculate an average value of pixel heights of the actual bounding boxes of the stable tracking targets in the current frame image to obtain a second average value; a fourth calculation unit configured to calculate a difference value between the first average value and the second average value; a second judgment unit configured to judge whether the difference value is greater than a preset value; a third output unit configured to output a judgment conclusion that the camera is in a jolt state when the difference value is greater than the preset value; and a fourth output unit configured to output a judgment conclusion that the camera is in a non-jolt state when the difference value is not greater than the preset value.

[0012] The third aspect of the application provides a target tracking method suitable for a camera in a jolt state, which uses the above-mentioned jolt state identification method to judge whether the target camera is in a jolt state; the target tracking method comprises: for tracking targets for which the estimated bounding box and the actual bounding box are not matched successfully, respectively enlarging the height of the estimated bounding box and the actual bounding box of the tracking target; calculating an intersection-over-union of the area of the enlarged estimated bounding box and the area of the enlarged actual bounding box; judging whether the intersection-over-union is greater than a threshold value; if yes, updating the estimated bounding box with the actual bounding box and outputting a tracking result of the tracking target; otherwise, abandoning the corresponding tracking target.

[0013] In some embodiments of the third aspect, the respectively enlarging the height of the estimated bounding box and the actual bounding box of the tracking target comprises: lengthening the top edge of the estimated bounding box upward by a preset box height and lengthening the bottom edge downward by the preset box height; lengthening the top edge of the actual bounding box upward by the preset box height and lengthening the bottom edge downward by the preset box height.

[0014] The fourth aspect of the present application provides a target tracking device suitable for a camera in a jolting state, which uses the above-mentioned jolting state identification method to determine whether the target camera is in a jolting state; the target tracking device comprises: an expansion module, configured to expand the height of an estimated bounding box and an actual bounding box of a tracking target, respectively, the tracking target being a tracking target for which the estimated bounding box and the actual bounding box are not successfully matched; a second calculation module, configured to calculate the intersection-over-union of the area of the expanded estimated bounding box and the area of the expanded actual bounding box; a judgment module, configured to determine whether the intersection-over-union is greater than a threshold value; a first execution module, configured to update the estimated bounding box with the actual bounding box when the intersection-over-union is greater than the threshold value, and output a tracking result of the tracking target; and a second execution module, configured to discard the corresponding tracking target when the intersection-over-union is not greater than the threshold value.

[0015] Based on the fourth aspect, in some embodiments of the present application, the expansion module comprises: a first expansion unit, configured to extend the top edge of the estimated bounding box upward by a preset frame height and extend the bottom edge downward by the preset frame height; and a second expansion unit, configured to extend the top edge of the actual bounding box upward by the preset frame height and extend the bottom edge downward by the preset frame height.

[0016] Based on the fourth aspect, in some embodiments of the present application, the preset frame height is 1 / 2 of the frame height of the estimated bounding box or the frame height of the actual bounding box.

[0017] The present application has at least the following beneficial effects:

[0018] 1) The jolting state identification method provided by the present application is implemented based on multi-target matching on the basis of existing target tracking methods. In the judgment process, first, the jolting state of the camera is preliminarily judged according to the quantity relationship between the total number of tracking targets and the number of matched targets, and when an uncertain situation occurs, a quantitative calculation and judgment verification method is further used to determine the jolting state of the camera, thereby ensuring the accuracy of the judgment and reducing the probability of target tracking failure caused by inaccurate jolting state judgment and the failure to take the correct tracking method in time.

[0019] 2) The target tracking method provided by the present application is mainly suitable for a camera in a jolting state, and the height range of the estimated bounding box and the actual bounding box is expanded to compensate for the up-down displacement of the tracking target in the shooting picture caused by the up-down jolting of the camera, thereby improving the matching probability of the estimated bounding box and the actual bounding box and improving the stability and robustness of target tracking. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 FIG. 1 is a flowchart of the jolting state identification method in Embodiment 1 of the present application;

[0021] Figure 2 FIG. 2 is a flowchart of the jolting state identification method in Embodiment 2 of the present application;

[0022] Figure 3 Flowchart of the target tracking method in Embodiment 6 of the present application;

[0023] Figure 4 Actual shooting scene effect diagram of the prior art (top);

[0024] Figure 5 Actual shooting scene effect diagram of the prior art (bottom);

[0025] Figure 6 Actual shooting scene effect diagram after using the target tracking method of the present application (top);

[0026] Figure 7 Actual shooting scene effect diagram after using the target tracking method of the present application (bottom). DETAILED DESCRIPTION

[0027] Other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the present specification. The present application can also be implemented or applied by means of other different specific embodiments, and various modifications or changes can be made to the details in the present specification based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, and are not intended to limit the protection scope of the present application.

[0028] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present application, and only the components related to the present application are shown in the diagrams, rather than being drawn according to the number, shape and size of the components in actual implementation. The type, number and proportion of the components in actual implementation can be arbitrarily changed, and the layout type of the components can also be more complex.

[0029] Embodiment 1

[0030] Please refer to Figure 1 The present embodiment provides a bump state recognition method suitable for a camera in a moving state (the judgment of the bump state of the camera in the present embodiment is based on the ground as a reference). The bump state recognition method comprises the following steps:

[0031] S101, collecting a current frame image and a historical frame image containing a tracking target;

[0032] Since the tracking target is multiple, the multiple tracking targets can not be in the same frame image, and therefore this step can be implemented only for a single tracking target.

[0033] S102, estimating an estimated bounding box of each tracking target in the current frame image based on the historical frame image;

[0034] The target on the historical frame track is predicted on the position of the target in the current frame image by using the time difference between two frames and the target speed calculated by tracking. The specific method is: selecting the bottom edge midpoint of the target frame of the tracking target as the ranging point of the target, using the camera calibration or pinhole imaging method to obtain the historical world position of the ranging point of the target, and then obtaining the historical estimated speed of the target through the known image timestamp, so as to predict the world coordinates of the ranging point of the tracking target in the current frame. The world coordinates of the ranging point are converted to image coordinates by using the camera calibration or pinhole imaging method again, and the predicted bounding box of the target can be obtained.

[0035] S103, obtaining the actual bounding box of each tracking target from the current frame image;

[0036] The detection model is used to detect the current frame image to obtain the actual bounding box of the tracking target.

[0037] S104, obtaining the number of matching between the predicted bounding box and the actual bounding box, and obtaining the matching number;

[0038] Specifically, the judgment method of whether the predicted bounding box matches the actual bounding box is as follows:

[0039] Calculating the intersection-over-union of the area of the predicted bounding box and the area of the actual bounding box in the current frame image;

[0040] Judging whether the intersection-over-union is greater than a threshold value;

[0041] If yes, the predicted bounding box of the tracking target matches the actual bounding box;

[0042] Otherwise, the predicted bounding box of the tracking target does not match the actual bounding box.

[0043] S105, if the matching number is zero, it is determined that the camera is in a jolt state;

[0044] Further, the total number of actual bounding boxes can also be obtained; if the matching number is the total number, it is determined that the camera is in a non-jolt state;

[0045] If the matching number is not zero but less than the total number, a preset verification method is used to further verify whether the camera is in a jolt state, which specifically includes:

[0046] A1, selecting stable tracking targets with a historical number of successful matches greater than a preset number from the tracking targets;

[0047] A2, calculating the average value of the pixel height of the predicted bounding box of each stable tracking target in the current frame image to obtain a first average value;

[0048] A3、calculate the average value of the pixel height of the actual bounding box of each stable tracking target in the current frame image, to obtain a second average value;

[0049] For example, the average value of the pixel height of the estimated bounding box can be the average value of the pixel height of the top edge of the estimated bounding box; and the average value of the pixel height of the actual bounding box can be the average value of the pixel height of the top edge of the actual bounding box. If there is only one estimated bounding box (actual bounding box) in the current frame image, the pixel height of the top edge of the estimated bounding box (actual bounding box) is the average value.

[0050] When the camera (or camera) does not vibrate, the camera always takes pictures on the same horizontal plane. In the case that the tracking target does not vibrate up and down, the position height (ignoring the size of the target) of the tracking target mapped in the picture should always be unchanged. When the camera vibrates, the shooting angle of the camera will certainly shift up and down, for example, when the camera shoots upward, the position of the tracking target mapped in the shooting image will certainly shift downward in the case that the tracking target does not shift up and down.

[0051] Considering that it is too tedious and time-consuming to compare one by one if there are multiple tracking targets that do not match successfully in the same current frame image, in this embodiment, the average (height) value is calculated for comprehensive judgment.

[0052] A4、calculate the difference between the first average value and the second average value;

[0053] A5、determine whether the difference is greater than a preset value;

[0054] If yes, the camera is in a vibrating state;

[0055] Otherwise, the camera is in a non-vibrating state.

[0056] Embodiment 2

[0057] For reference Figure 2 , the embodiment provides another vibration state recognition method, which comprises:

[0058] S201、collect a current frame image and a historical frame image containing a tracking target;

[0059] S202、estimate the estimated bounding box of each tracking target in the current frame image based on the historical frame image;

[0060] S203、obtain the actual bounding box of each tracking target from the current frame image;

[0061] For the specific implementation process of steps S201-S203, reference can be made to steps S101-S103 in embodiment 1.

[0062] S204, obtain the number of tracking targets whose estimated bounding box matches the actual bounding box, to obtain a matching number;

[0063] Unlike directly counting the number of matching bounding boxes in Embodiment 1, in this embodiment, the number of tracking targets is counted based on the mapping relationship between the bounding boxes (estimated bounding box and actual bounding box) and the tracking targets. This step is suitable for the case when there are multiple estimated bounding boxes.

[0064] S205, obtain the total number of tracking targets;

[0065] That is, the number of tracking targets is obtained. In this embodiment, multiple tracking targets are taken as an example (more than 2).

[0066] S206, calculate the difference between the total number and the matching number;

[0067] Since the matching number of tracking targets cannot be greater than the total number of tracking targets, there are the following three cases between the total number and the matching number.

[0068] If the difference is zero, it is determined that the camera is in a non-turbulent state;

[0069] If the difference is the total number value, it is determined that the camera is in a turbulent state;

[0070] If the difference is greater than zero and less than the total number value, a preset verification method is used to further verify whether the camera is in a turbulent state.

[0071] The fact that the estimated bounding box does not match the actual bounding box may not be that the camera has been tilted up and down, but that the camera has been displaced in the horizontal direction, so further verification is needed. The specific preset verification method can be referred to in Embodiment 1.

[0072] Embodiment 3

[0073] This embodiment provides a turbulent state recognition device suitable for a camera in a moving state. The turbulent state recognition device comprises: a collection module for collecting a current frame image and a historical frame image containing tracking targets; an estimation module for estimating an estimated bounding box of each tracking target in the current frame image based on the historical frame image; a first acquisition module for acquiring an actual bounding box of each tracking target from the current frame image; a second acquisition module for obtaining the number of tracking targets whose estimated bounding box matches the actual bounding box, to obtain a matching number; a third acquisition module for obtaining the total number of tracking targets; a first calculation module for calculating the difference between the total number and the matching number; and a first output module for outputting a recognition result that the camera is in a turbulent state when the matching number is zero.

[0074] Further, the second obtaining module comprises: a judging unit group, configured to judge whether the estimated bounding box matches the actual bounding box; the judging unit group comprises: a first calculating unit, configured to calculate an intersection-over-union of the area of the estimated bounding box and the area of the actual bounding box in the current frame image; a first judging unit, configured to judge whether the intersection-over-union is greater than a threshold value; a first output unit, configured to output a judgment conclusion that the estimated bounding box of the tracking target matches the actual bounding box when the intersection-over-union is greater than the threshold value; and a second output unit, configured to output a judgment conclusion that the estimated bounding box of the tracking target does not match the actual bounding box when the intersection-over-union is not greater than the threshold value.

[0075] Further, the device for identifying the state of jolting further comprises: a third obtaining module, configured to obtain a total number of actual bounding boxes; and a verifying module, configured to further verify whether the camera is in the state of jolting by using a preset verifying method when the matching number is not zero but less than the total number; the verifying module comprises: a selecting unit, configured to select stable tracking targets with a number of historical matching successes greater than a preset number from the tracking targets; a second calculating unit, configured to calculate an average value of the pixel height of the estimated bounding box of each stable tracking target in the current frame image to obtain a first average value; a third calculating unit, configured to calculate an average value of the pixel height of the actual bounding box of each stable tracking target in the current frame image to obtain a second average value; a fourth calculating unit, configured to calculate a difference value between the first average value and the second average value; a second judging unit, configured to judge whether the difference value is greater than a preset value; a third output unit, configured to output a judgment conclusion that the camera is in the state of jolting when the difference value is greater than the preset value; and a fourth output unit, configured to output a judgment conclusion that the camera is in the state of not jolting when the difference value is not greater than the preset value.

[0076] The device provided in this embodiment can be used to implement the method described in Embodiment 1, which is specifically shown in Embodiment 1 and will not be repeated here.

[0077] Embodiment 4

[0078] This embodiment provides a target tracking method, which is suitable for a camera in a state of jolting, and judges whether the target camera is in the state of jolting by using the method described in Embodiment 1; the target tracking method comprises: for tracking targets for which the estimated bounding box does not match the actual bounding box, respectively enlarging the height of the estimated bounding box and the height of the actual bounding box of the tracking target; calculating an intersection-over-union of the area of the enlarged estimated bounding box and the area of the enlarged actual bounding box; judging whether the intersection-over-union is greater than a threshold value; if yes, updating the estimated bounding box by using the actual bounding box, and outputting a tracking result of the tracking target; otherwise, abandoning the corresponding tracking target.

[0079] Further, the method for respectively expanding the height of the estimated bounding box and the actual bounding box of the tracking target comprises: lengthening the top edge of the estimated bounding box upward by 1 / 2 of the height of the box and lengthening the bottom edge downward by 1 / 2 of the height of the box; lengthening the top edge of the actual bounding box upward by 1 / 2 of the height of the box and lengthening the bottom edge downward by 1 / 2 of the height of the box. That is, the height of the estimated bounding box and the actual bounding box is expanded by one time. If the corresponding target still cannot be tracked in this case, it means that the target tracking fails, and at this time, the target is abandoned and a new tracking target is determined.

[0080] Since the target matching method based on the intersection over union of the target box cannot well perform target matching in the jolt state, in this embodiment, whether the jolt state is determined by using the result of the target matching in Embodiment 1, to determine whether the re-matching in the jolt state is needed. When it is determined that the camera is in the jolt state, the height range of the bounding box (detection box) is expanded, which can effectively avoid the problem of target tracking failure caused by the jolt, and improve the stability and robustness of the target tracking method.

[0081] Embodiment 5

[0082] The embodiment provides a target tracking device suitable for a camera in a jolt state, and the method in Embodiment 1 is used to determine whether the target camera is in the jolt state. The target tracking device comprises: an expansion module, which is used to respectively expand the height of an estimated bounding box and an actual bounding box of a tracking target, the tracking target being a tracking target for which the estimated bounding box and the actual bounding box are not matched successfully; a second calculation module, which is used to calculate the intersection over union of the area of the expanded estimated bounding box and the area of the expanded actual bounding box; a judgment module, which is used to determine whether the intersection over union is greater than a threshold value; a first execution module, which is used to update the estimated bounding box with the actual bounding box when the intersection over union is greater than the threshold value, and output a tracking result of the tracking target; and a second execution module, which is used to abandon the corresponding tracking target when the intersection over union is not greater than the threshold value.

[0083] Further, the expansion module comprises: a first expansion unit, which is used to lengthen the top edge of the estimated bounding box upward by 1 / 2 of the height of the box and lengthen the bottom edge downward by 1 / 2 of the height of the box; and a second expansion unit, which is used to lengthen the top edge of the actual bounding box upward by 1 / 2 of the height of the box and lengthen the bottom edge downward by 1 / 2 of the height of the box.

[0084] Embodiment 6

[0085] As Figure 3The shown is a flowchart of a method combining bump state recognition and target tracking. Specifically, the actual target set and the estimated target set are obtained, the image is collected by the camera, and then the target detection algorithm is used to identify the target from the image to obtain the actual target set. The estimated target set is obtained by the historical target of the previous historical frame according to the trajectory history information. The specific method is: selecting the midpoint of the bottom edge of the target frame as the ranging point of the target, using camera calibration or pinhole imaging method to obtain the historical world position of the ranging point of the target, and then through the known image timestamp, the historical estimated speed of the target can be obtained, so that the world coordinates of the ranging point of the historical target in the current frame can be estimated. Again, the camera calibration or pinhole imaging method is used to convert the world coordinates of the ranging point to image coordinates, and the estimated frame of the target can be obtained. Then the intersection-over-union of the actual target and the estimated target is calculated, and the maximum intersection-over-union is selected as the matching successful target pair.

[0086] There are three cases of target matching results obtained. First, in the case where the number of estimated targets is greater than zero, if none of them is matched, it is considered to be a bump state. Second, in the case where the number of estimated targets is greater than zero, the number of matched targets and estimated targets is the same, which is not a bump state. Third, in the case where the number of estimated targets is greater than zero, if there are matched targets but less than the number of estimated targets, it may be in a bump state and needs to be further verified.

[0087] The verification method is specifically implemented as follows: first, select the target that is currently successfully matched and has a long history tracking time from the previously obtained target matching results as a stable target. Then, calculate the sum of the heights of the top edges of all stable target frames and obtain the average value, and similarly obtain the average height of the top edges of the estimated target frame. Finally, calculate the difference between the two average values. If the difference is greater than a set threshold, it is considered to be a bump state; if the difference is less than a set threshold, it is considered to be a non-bump state.

[0088] After obtaining the bump state, if it is determined to be non-bump, it is considered that the previously matched target pair is valid, and the target is updated according to the matching result; if it is determined to be bump, the height of the actual target frame and the estimated target frame is enlarged, the specific method is to extend the top edge upward by 1 / 2 frame height and the bottom edge downward by 1 / 2 frame height, and then recalculate the intersection-over-union of the target frames between the targets to obtain the matched target pair. The target is updated according to the matching result.

[0089] Taking the use scene of a front camera of a car as an example, Figure 4 is an effect diagram of the shooting scene of the previous frame, in which the tracking position of the tracking target is marked, Figure 5The effect picture of the current frame using the target frame-based IOU target tracking method under the slight shaking of the camera, obviously, when the target is far away, the slight shaking of the camera leads to the failure of tracking the target (the target frame and the position data of the target frame do not appear in the picture).

[0090] In the same scene, Figure 6 The effect picture of the last frame, the tracking target is consistent with the tracking target in Figure 4 Figure 7 The effect picture of the current frame after using the tracking method of the present application, obviously, even if the target is far away and the camera slightly shakes, the target can still be successfully tracked.

[0091] The device provided by the embodiment can be used to implement the method described in embodiment 4, which is specifically shown in embodiment 4, and will not be repeated here.

[0092] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a device, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program code.

[0093] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), apparatuses and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0094] These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing apparatus to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0095] ​These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1

[0096] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0097] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), flash memory, or a combination of non-volatile memories in different types. The memory can also include a compact disk read only memory (CD-ROM), digital versatile disk (DVD), Blu-ray, or another non-transitory computer readable medium, which is non-volatile and non-transitory in nature, but volatile in that it can lose its content if the power to the computer is turned off or if the computer crashes. The memory is an example of a computer readable medium.

[0098] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media such as modulated data signals and carriers.

[0099] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0100] ​​The above merely illustrates the embodiments of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. within the spirit and principles of the present application shall fall into the scope of claims of the present application.

Claims

1. A method for recognizing bumpy conditions, applicable to cameras in motion, characterized in that, The method for identifying the jitter state comprises: collecting a current frame image and a historical frame image containing tracking targets; estimating an estimated bounding box of each tracking target in the current frame image based on the historical frame image; obtaining an actual bounding box of each tracking target from the current frame image; obtaining a matching number of the estimated bounding box and the actual bounding box, and obtaining the matching number; if the matching number is zero, determining that the camera is in a jitter state; obtaining a total number of the actual bounding boxes; if the matching number is not zero but less than the total number, further verifying whether the camera is in a jitter state by using a preset verification method, specifically comprising: selecting stable tracking targets with a historical matching success number greater than a preset number from the tracking targets; calculating an average value of pixel heights of the estimated bounding boxes of the stable tracking targets in the current frame image, and obtaining a first average value; calculating an average value of pixel heights of the actual bounding boxes of the stable tracking targets in the current frame image, and obtaining a second average value; calculating a difference value of the first average value and the second average value; when the difference value is greater than a preset value, determining that the camera is in a jitter state.

2. The surge state recognition method according to claim 1, characterized by, The method for determining whether the estimated bounding box and the actual bounding box match is as follows: calculating an intersection-over-union of the area of the estimated bounding box and the area of the actual bounding box in the current frame image; determining whether the intersection-over-union is greater than a threshold value; if yes, determining that the estimated bounding box and the actual bounding box of the tracking target match; otherwise, determining that the estimated bounding box and the actual bounding box of the tracking target do not match.

3. A jolt state recognition device adapted for a camera in a moving state, characterized by, The device for identifying the jitter state comprises: a collecting module configured to collect a current frame image and a historical frame image containing tracking targets; an estimating module configured to estimate an estimated bounding box of each tracking target in the current frame image based on the historical frame image; a first obtaining module configured to obtain an actual bounding box of each tracking target from the current frame image; a second obtaining module configured to obtain a matching number of the estimated bounding box and the actual bounding box, and obtain the matching number; a first output module configured to output a recognition result that the camera is in a jitter state when the matching number is zero; a third obtaining module configured to obtain a total number of the actual bounding boxes; a verification module configured to further verify whether the camera is in a jitter state by using a preset verification method when the matching number is not zero but less than the total number; the verification module comprises: a selecting unit configured to select stable tracking targets with a historical matching success number greater than a preset number from the tracking targets; a second calculating unit configured to calculate an average value of pixel heights of the estimated bounding boxes of the stable tracking targets in the current frame image, and obtain a first average value; a third calculating unit configured to calculate an average value of pixel heights of the actual bounding boxes of the stable tracking targets in the current frame image, and obtain a second average value; a fourth calculating unit configured to calculate a difference value of the first average value and the second average value; a second judging unit configured to determine whether the difference value is greater than a preset value; a third output unit configured to output a judgment result that the camera is in a jitter state when the difference value is greater than the preset value.

4. The jerk state recognition device according to claim 3, characterized by The second obtaining module comprises: a judging unit group configured to determine whether the estimated bounding box and the actual bounding box match. The judging unit group comprises: A first calculating unit configured to calculate an intersection-over-union of an estimated bounding box area and an actual bounding box area in a current frame image; A first judging unit configured to judge whether the intersection-over-union is greater than a threshold value; A first output unit configured to output a judgment result that the estimated bounding box of the tracking target matches the actual bounding box when the intersection-over-union is greater than the threshold value; A second output unit configured to output a judgment result that the estimated bounding box of the tracking target does not match the actual bounding box when the intersection-over-union is not greater than the threshold value.

5. A target tracking method, suitable for a camera in a jolting state, characterized in that, The method for identifying a bumping state according to any one of claims 1-2 is used to judge whether a target camera is in a bumping state; The target tracking method comprises: For a tracking target whose estimated bounding box and actual bounding box do not match successfully, the height of the estimated bounding box and the height of the actual bounding box of the tracking target are respectively enlarged; An intersection-over-union of the area of the enlarged estimated bounding box and the area of the enlarged actual bounding box is calculated; It is judged whether the intersection-over-union is greater than a threshold value; If yes, the estimated bounding box is updated with the actual bounding box, and a tracking result of the tracking target is outputted; Otherwise, the corresponding tracking target is abandoned.

6. The target tracking method according to claim 5, characterized by, The height of the estimated bounding box and the height of the actual bounding box of the tracking target are respectively enlarged, comprising: The top edge of the estimated bounding box is extended upwards by a preset frame height, and the bottom edge is extended downwards by the preset frame height; The top edge of the actual bounding box is extended upwards by the preset frame height, and the bottom edge is extended downwards by the preset frame height.

7. A target tracking device adapted for a camera in a jolting state, characterized in that, The device for identifying a bumping state according to any one of claims 3-4 is used to judge whether a target camera is in a bumping state; The target tracking device comprises: An enlarging module configured to respectively enlarge the height of the estimated bounding box and the height of the actual bounding box of a tracking target, the tracking target being a tracking target whose estimated bounding box and actual bounding box do not match successfully; A second calculating module configured to calculate an intersection-over-union of the area of the enlarged estimated bounding box and the area of the enlarged actual bounding box; A judging module configured to judge whether the intersection-over-union is greater than a threshold value; A first executing module configured to, when the intersection-over-union is greater than the threshold value, update the estimated bounding box with the actual bounding box, and output a tracking result of the tracking target; A second executing module configured to, when the intersection-over-union is not greater than the threshold value, abandon the corresponding tracking target.

8. The object tracking apparatus according to claim 7, characterized by The enlarging module comprises: A first enlarging unit configured to extend the top edge of the estimated bounding box upwards by a preset frame height, and extend the bottom edge downwards by the preset frame height; A second enlarging unit configured to extend the top edge of the actual bounding box upwards by the preset frame height, and extend the bottom edge downwards by the preset frame height.

Images