Target matching method and device and electronic equipment

Abstract

The embodiment of the invention provides a target matching method and device and electronic equipment, and the method comprises the steps: responding to a first target which is detected by a gun camera and meets a preset alarm condition, calculating a PTZ coordinate enabling the position of the first target to be located in the visual field of a dome camera, and controlling the dome camera to move to the determined PTZ coordinate; controlling the gun camera to shoot to obtain a plurality of first image frames, and controlling the dome camera to synchronously shoot to obtain a plurality of second image frames in a static state; detecting to obtain a second target existing in each second image frame, and determining whether a third target meeting a preset matching condition exists in each second target; wherein the preset matching condition comprises that the similarity degree between the second motion track corresponding to the second target and the first motion track is greater than the first preset similarity degree. By applying the technical scheme provided by the embodiment of the invention, the target detected by the gun camera can be accurately determined from the field of view of the dome camera.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a target matching method, device and electronic equipment. Background Art

[0002] In order to meet the increasingly complex monitoring scenarios, the use of gun-ball linkage solutions (referred to as gun-ball linkage) is becoming more and more popular. The gun-ball linkage solution usually includes a gun camera and at least one ball camera, in which the gun camera is responsible for detecting the target, and then, according to the position of the target in the image obtained by the gun camera, the ball camera is controlled to move in the PTZ coordinate system, so that the ball camera can clearly capture the target detected by the gun camera, and then the ball camera can follow the target detected by the gun camera, determine the details, and other operations.

[0003] It can be seen that in the gun-ball linkage solution, the target detected by the gun camera must be determined in the field of view of the ball camera before the subsequent steps can be implemented. However, when the ball camera moves in the PTZ coordinate system according to the position of the target in the gun camera, there may be multiple targets in the field of view of the ball camera. At this time, if it is not possible to accurately determine which specific target in the field of view of the ball camera is the target detected by the gun camera, the subsequent steps cannot be accurately executed. For example, the ball camera is required to follow the target detected by the gun camera. When the ball camera moves in the PTZ coordinate system according to the position of the target of interest in the gun camera, there are multiple targets in the field of view of the ball camera. At this time, if a certain target in the field of view of the ball camera (hereinafter referred to as the misidentified target) is mistakenly confirmed as the target detected by the gun camera, then the subsequent following of the misidentified target is obviously inaccurate, and if other judgments are made based on the following results, the judgment results will also be inaccurate. Based on this, it is crucial to accurately determine the target detected by the gun camera from the field of view of the ball camera. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a target matching method, device and electronic device to accurately determine the target detected by the gun camera from the field of view of the ball camera. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a target matching method, the method comprising:

[0006] In response to the gun camera detecting the first target, calculating PTZ coordinates that can make the position of the first target within the field of view of the ball camera, controlling the ball camera to move to the determined PTZ coordinates, and controlling the ball camera to remain stationary after moving to the determined PTZ coordinates;

[0007] After the ball camera moves to the determined PTZ coordinates, the gun camera is controlled to shoot to obtain a plurality of first image frames, and the ball camera is controlled to synchronously shoot to obtain a plurality of second image frames in a stationary state, and a timestamp of each of the first image frames and each of the second image frames is obtained;

[0008] Detect and obtain the second target existing in each of the second image frames, and determine whether there is a third target satisfying a preset matching condition in each of the second targets; wherein the preset matching condition includes: the degree of similarity between the second motion trace corresponding to the second target and the first motion trace is greater than a first preset degree of similarity; the first motion trace is the motion trace of the first target determined based on each of the first image frames and the timestamps of each of the first image frames; the second motion trace is the motion trace of the second target determined based on each of the second image frames and the timestamps of each of the second image frames.

[0009] In some embodiments, the first target is a potential target that meets a preset alarm condition, and the method further includes:

[0010] If there is a third target, determining that the first target is a target that meets the preset alarm condition;

[0011] If the third target does not exist, the first target is determined to be a target that does not meet the preset alarm condition.

[0012] In some embodiments, the similarity between each of the second motion traces and the first motion trace is determined in the following manner:

[0013] Determine the minimum circumscribed rectangle of the first motion trace as a first circumscribed rectangle; and respectively determine the minimum circumscribed rectangle of each of the second motion traces as a second circumscribed rectangle;

[0014] According to the first circumscribed rectangle, each track point in the first motion track is scaled to obtain a third motion track; and according to each of the second circumscribed rectangles, each track point in each of the second motion tracks is scaled to obtain a fourth motion track corresponding to each of the second motion tracks; wherein each of the fourth motion tracks has the same size as the third motion track;

[0015] The similarity between each of the fourth motion traces and the third motion trace is calculated as the similarity between each of the second motion traces and the first motion trace.

[0016] In some embodiments, scaling each track point in the first motion track according to the first circumscribed rectangle to obtain a third motion track; and scaling each track point in each second motion track according to each second circumscribed rectangle to obtain a fourth motion track corresponding to each second motion track, includes:

[0017] Determine the coordinates of the vertex of the upper left corner of the first circumscribed rectangle, and the width and height of the first circumscribed rectangle; and respectively determine the coordinates of the vertex of the upper left corner of each of the second circumscribed rectangles, and the width and height of each of the second circumscribed rectangles;

[0018] The coordinates of the trace points constituting the third motion trace and the coordinates of the trace points constituting each of the fourth motion traces are determined by the following formula;

[0019]

[0020]

[0021] Among them, when ( , ) is the coordinate of the i-th track point in the first motion track, ( , ) is the coordinate of the vertex of the upper left corner of the first circumscribed rectangle, is the width of the first circumscribed rectangle, is the height of the first circumscribed rectangle, ( , ) is the coordinate of the i-th track point in the third motion track; when ( , ) is the coordinate of the i-th track point in the second motion track, ( , ) is the coordinate of the upper left corner of the second circumscribed rectangle, is the width of the second circumscribed rectangle, is the height of the second circumscribed rectangle, ( , ) are the coordinates of the i-th track point in the fourth motion track.

[0022] In some embodiments, the preset matching condition further includes that the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity;

[0023] The movement direction of the first target is the movement direction of the first target during a period from the first first image frame obtained by photographing the first target to the last first image frame;

[0024] The movement direction of each second target is the movement direction of the second target to which the second movement track belongs during a period from a first second image frame obtained by photographing the second target to which the second movement track belongs to to a last second image frame.

[0025] In some embodiments, the movement direction of the first target is determined by:

[0026] Determine the coordinates of the first target in a first first image frame obtained by photographing the first target as the first coordinates; and determine the coordinates of the first target in a last first image frame obtained by photographing the first target as the second coordinates;

[0027] Calculate the direction of the straight line formed by the first coordinate and the second coordinate according to the first coordinate and the second coordinate as the moving direction of the first target;

[0028] For each of the second targets, the movement direction of the second target is determined by:

[0029] Determine the coordinates of the second target in the first second image frame obtained by photographing the second target as the third coordinates; and determine the coordinates of the second target in the last second image frame obtained by photographing the second target as the fourth coordinates;

[0030] The direction of the straight line formed by the third coordinate and the fourth coordinate is calculated according to the third coordinate and the fourth coordinate as the moving direction of the second target to which the second moving track belongs.

[0031] In some embodiments, the preset matching condition further includes that the distance between the position of the second target and the position of the first target is less than a preset distance;

[0032] The distance between the position of each second target and the position of the first target is determined by:

[0033] Selecting at least one image frame from the first image frame as a third image frame; and determining an image frame corresponding to the third image frame from the second image frame as a fourth image frame corresponding to the third image frame, wherein the fourth image frame is an image frame with the same timestamp as the third image frame;

[0034] For each of the third image frames, projecting the position of the first target in the third image frame and the position of each of the second targets in the fourth image frame corresponding to the third image frame into a target coordinate system;

[0035] In the target coordinate system, the distance between the position of each second target after projection and the position of the first target after projection is calculated to obtain the distance between the position of each second target and the position of the first target.

[0036] In some embodiments, the preset matching condition further includes that a size difference between the size of the second object and the size of the first object is less than a preset difference;

[0037] The size of the first target and the size of each of the second targets are determined by:

[0038] Selecting a preset number of image frames from the first image frames as fifth image frames, and selecting the preset number of image frames from the second image frames as sixth image frames;

[0039] For the fifth image frame and the sixth image frame, determine the size of the first object in the fifth image frame, and determine the size of each of the second objects in the sixth image frame;

[0040] The difference between the size of the first object and the size of each of the second objects is calculated to obtain the size difference between the size of each of the second objects and the size of the first object.

[0041] In some embodiments, determining the size of the first object in the fifth image frame includes:

[0042] According to the coordinate conversion relationship between the coordinate system of the fifth image frame and the PTZ coordinate system, the four vertex coordinates of the first target frame in the fifth image frame are mapped to the PTZ coordinate system to obtain the projected vertex coordinates; wherein the first target frame is the target frame where the first target is located when the first target in the fifth image frame is identified;

[0043] Determine the size of the first object according to the width and height of the area enclosed by the projected vertex coordinates;

[0044] The determining the size of each of the second objects in the sixth image frame includes:

[0045] The size of each second target is determined according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame.

[0046] In some embodiments, the size of the first object is determined according to the projected vertex coordinates by the following formula:

[0047] pan_field L =(pan right_top +pan right_bottom -pan left_top -pan left_bottom ) / 2

[0048] tilt_fieldL =(tilt right_top +tilt left_top -tilt right_bottom -tilt left_bottom ) / 2

[0049] Among them, pan_field L The width of the first target; tilt_field L is the height of the first target; (pan right_top , tilt right_top ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper right corner of the first target frame; (pan right_bottom , tilt right_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the lower right corner of the first target frame; (pan left_top , pan left_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper left corner of the first target frame; (pan left_bottom , tilt left_bottom ) is the projected vertex coordinates obtained by projecting the coordinates of the vertex at the lower left corner of the first target frame;

[0050] The size of each second target is determined by the following formula according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame:

[0051] pan_field F =pan_field zoom ×h target / h FOV

[0052] tilt_field F =tilt_field zoom × target / w FOV

[0053] Among them, pan_field F The width of the second target; tilt_field F is the height of the second target; (pan_field zoom , tilt_field zoom ) is the field of view of the ball camera; (h target , w target ) is the width and height of the area occupied by the second target in the sixth image frame; (h FOV , wFOV ) are the width and height of the sixth image frame.

[0054] In some embodiments, the preset matching conditions also include: the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity, the distance between the position of the second target and the position of the first target is less than a preset distance, and the size difference between the size of the second target and the size of the first target is less than a preset difference.

[0055] In some embodiments, the lens in the gun camera is a thermal imaging lens, and the lens in the ball camera is a visible light lens.

[0056] In a second aspect, an embodiment of the present application provides a target matching device, the device comprising:

[0057] A motion control module, configured to calculate PTZ coordinates that enable the position of the first target to be located within the field of view of the ball camera in response to the gun camera detecting the first target, control the ball camera to move to the determined PTZ coordinates, and control the ball camera to remain stationary after moving to the determined PTZ coordinates;

[0058] An image acquisition module, used for controlling the gun camera to shoot a plurality of first image frames after the ball camera moves to the determined PTZ coordinates, and controlling the ball camera to synchronously shoot a plurality of second image frames in a stationary state, and obtaining a timestamp of each of the first image frames and each of the second image frames;

[0059] A target determination module is used to detect the second target existing in each of the second image frames, and determine whether there is a third target that meets the preset matching conditions in each of the second targets; wherein the preset matching conditions include: the degree of similarity between the second motion trace corresponding to the second target and the first motion trace is greater than a first preset degree of similarity; the first motion trace is the motion trace of the first target determined based on each of the first image frames and the timestamps of each of the first image frames; the second motion trace is the motion trace of the second target determined based on each of the second image frames and the timestamps of each of the second image frames.

[0060] In some embodiments, the first target is a potential target that meets a preset alarm condition, and the device further includes:

[0061] The alarm determination module is used to determine that the first target is a target that meets the preset alarm condition if a third target exists; if the third target does not exist, determine that the first target is a target that does not meet the preset alarm condition.

[0062] In some embodiments, the similarity between each of the second motion traces and the first motion trace is determined in the following manner:

[0063] Determine the minimum circumscribed rectangle of the first motion trace as a first circumscribed rectangle; and respectively determine the minimum circumscribed rectangle of each of the second motion traces as a second circumscribed rectangle;

[0064] According to the first circumscribed rectangle, each track point in the first motion track is scaled to obtain a third motion track; and according to each of the second circumscribed rectangles, each track point in each of the second motion tracks is scaled to obtain a fourth motion track corresponding to each of the second motion tracks; wherein each of the fourth motion tracks has the same size as the third motion track;

[0065] The similarity between each of the fourth motion traces and the third motion trace is calculated as the similarity between each of the second motion traces and the first motion trace.

[0066] In some embodiments, scaling each track point in the first motion track according to the first circumscribed rectangle to obtain a third motion track; and scaling each track point in each second motion track according to each second circumscribed rectangle to obtain a fourth motion track corresponding to each second motion track, includes:

[0067] Determine the coordinates of the vertex of the upper left corner of the first circumscribed rectangle, and the width and height of the first circumscribed rectangle; and respectively determine the coordinates of the vertex of the upper left corner of each of the second circumscribed rectangles, and the width and height of each of the second circumscribed rectangles;

[0068] The coordinates of the trace points constituting the third motion trace and the coordinates of the trace points constituting each of the fourth motion traces are determined by the following formula;

[0069]

[0070]

[0071] Among them, when ( , ) is the coordinate of the i-th track point in the first motion track, ( , ) is the coordinate of the vertex of the upper left corner of the first circumscribed rectangle, is the width of the first circumscribed rectangle, is the height of the first circumscribed rectangle, ( , ) is the coordinate of the i-th track point in the third motion track; when ( , ) is the coordinate of the i-th track point in the second motion track, ( , ) is the coordinate of the upper left corner of the second circumscribed rectangle, is the width of the second circumscribed rectangle, is the height of the second circumscribed rectangle, ( , ) are the coordinates of the i-th track point in the fourth motion track.

[0072] In some embodiments, the preset matching condition further includes that the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity;

[0073] The movement direction of the first target is the movement direction of the first target during a period from the first first image frame obtained by photographing the first target to the last first image frame;

[0074] The movement direction of each second target is the movement direction of the second target to which the second movement track belongs during a period from a first second image frame obtained by photographing the second target to which the second movement track belongs to to a last second image frame.

[0075] In some embodiments, the movement direction of the first target is determined by:

[0076] Determine the coordinates of the first target in a first first image frame obtained by photographing the first target as the first coordinates; and determine the coordinates of the first target in a last first image frame obtained by photographing the first target as the second coordinates;

[0077] Calculate the direction of the straight line formed by the first coordinate and the second coordinate according to the first coordinate and the second coordinate as the moving direction of the first target;

[0078] For each of the second targets, the movement direction of the second target is determined by:

[0079] Determine the coordinates of the second target in the first second image frame obtained by photographing the second target as the third coordinates; and determine the coordinates of the second target in the last second image frame obtained by photographing the second target as the fourth coordinates;

[0080] The direction of the straight line formed by the third coordinate and the fourth coordinate is calculated according to the third coordinate and the fourth coordinate as the moving direction of the second target to which the second moving track belongs.

[0081] In some embodiments, the preset matching condition further includes that the distance between the position of the second target and the position of the first target is less than a preset distance;

[0082] The distance between the position of each second target and the position of the first target is determined by:

[0083] Selecting at least one image frame from the first image frame as a third image frame; and determining an image frame corresponding to the third image frame from the second image frame as a fourth image frame corresponding to the third image frame, wherein the fourth image frame is an image frame with the same timestamp as the third image frame;

[0084] For each of the third image frames, projecting the position of the first target in the third image frame and the position of each of the second targets in the fourth image frame corresponding to the third image frame into a target coordinate system;

[0085] In the target coordinate system, the distance between the position of each second target after projection and the position of the first target after projection is calculated to obtain the distance between the position of each second target and the position of the first target.

[0086] In some embodiments, the preset matching condition further includes that a size difference between the size of the second object and the size of the first object is less than a preset difference;

[0087] The size of the first target and the size of each of the second targets are determined by:

[0088] Selecting a preset number of image frames from the first image frames as fifth image frames, and selecting the preset number of image frames from the second image frames as sixth image frames;

[0089] For the fifth image frame and the sixth image frame, determine the size of the first object in the fifth image frame, and determine the size of each of the second objects in the sixth image frame;

[0090] The difference between the size of the first object and the size of each of the second objects is calculated to obtain the size difference between the size of each of the second objects and the size of the first object.

[0091] In some embodiments, determining the size of the first object in the fifth image frame includes:

[0092] According to the coordinate conversion relationship between the coordinate system of the fifth image frame and the PTZ coordinate system, the four vertex coordinates of the first target frame in the fifth image frame are mapped to the PTZ coordinate system to obtain the projected vertex coordinates; wherein the first target frame is the target frame where the first target is located when the first target in the fifth image frame is identified;

[0093] Determine the size of the first object according to the width and height of the area enclosed by the projected vertex coordinates;

[0094] The determining the size of each of the second objects in the sixth image frame includes:

[0095] The size of each second target is determined according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame.

[0096] In some embodiments, the size of the first object is determined according to the projected vertex coordinates by the following formula:

[0097] pan_field L =(pan right_top +pan right_bottom -pan left_top -pan left_bottom ) / 2

[0098] tilt_field L =( tilt right_top + tilt left_top - tilt right_bottom - tilt left_bottom ) / 2

[0099] Among them, pan_field L The width of the first target; tilt_field L is the height of the first target; (pan right_top , tilt right_top ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper right corner of the first target frame; (pan right_bottom , tilt right_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the lower right corner of the first target frame; (pan left_top , pan left_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper left corner of the first target frame; (panleft_bottom , tilt left_bottom ) is the projected vertex coordinates obtained by projecting the coordinates of the vertex at the lower left corner of the first target frame;

[0100] The size of each second target is determined by the following formula according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame:

[0101] pan_field F =pan_field zoom ×h target / h FOV

[0102] tilt_field F =tilt_field zoom × target / w FOV

[0103] Among them, pan_field F The width of the second target; tilt_field F is the height of the second target; (pan_field zoom , tilt_field zoom ) is the field of view of the ball camera; (h target , w target ) is the width and height of the area occupied by the second target in the sixth image frame; (h FOV , w FOV ) are the width and height of the sixth image frame.

[0104] In some embodiments, the preset matching conditions also include: the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity, the distance between the position of the second target and the position of the first target is less than a preset distance, and the size difference between the size of the second target and the size of the first target is less than a preset difference.

[0105] In some embodiments, the lens in the gun camera is a thermal imaging lens, and the lens in the ball camera is a visible light lens.

[0106] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0107] Memory, used to store computer programs;

[0108] The processor is used to implement any of the above-mentioned target matching methods when executing the program stored in the memory.

[0109] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the program is executed by a processor, any of the above-mentioned target matching methods is implemented.

[0110] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the target matching methods described above.

[0111] Beneficial effects of the embodiments of the present application:

[0112] In the technical solution provided by the embodiment of the present application, in response to the gun camera detecting the first target, the PTZ coordinates that can make the position of the first target within the field of view of the ball camera are calculated, and then, after the ball camera is controlled to move to the determined PTZ coordinates, the first target is located within the field of view of the ball camera, and the ball camera can shoot the first target; the gun camera is controlled to shoot to obtain multiple first image frames, and the ball camera is controlled to synchronously shoot in a static state to obtain multiple second image frames, and the timestamps of each first image frame and each second image frame are obtained; then, the second target existing in each second image frame is detected, and it is determined whether there is a third target that meets the preset matching condition in each second target. Among them, the preset matching condition includes that the similarity between the second motion track corresponding to the second target and the first motion track is greater than the first preset similarity; and the first motion track is the motion track of the first target determined according to each first image frame and the timestamp of each first image frame; the second motion track is the motion track of the second target determined according to each second image frame and the timestamp of each second image frame.

[0113] It can be understood that for the gun-ball linkage scene, usually, the gun camera and the ball camera are shooting at a distance from the target, so the gun camera and the ball camera can be approximately regarded as being located at the same position. Therefore, for the same target, in the same time period, the shapes of its motion traces in the first image frame (i.e., the first motion trace) and the motion traces in the second image frame (i.e., the second motion trace) should be similar. Therefore, if the second motion trace corresponding to the second target is more similar to the first motion trace than the first preset similarity, it indicates that the second target is the third target, that is, the target that matches the first target detected by the gun camera. In this way, the target detected by the gun camera can be accurately determined from the field of view of the ball camera.

[0114] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0116] Figure 1 A schematic diagram of the structure of a gun-ball linkage device provided in an embodiment of the present application;

[0117] Figure 2 A schematic diagram of a flow chart of a target matching method provided in an embodiment of the present application;

[0118] Figure 3 A schematic diagram of a flow chart for determining the similarity of motion traces provided in an embodiment of the present application;

[0119] Figure 4 A schematic diagram of a motion trajectory and a minimum circumscribed rectangle provided in an embodiment of the present application;

[0120] Figure 5 A schematic diagram of the movement direction provided in an embodiment of the present application;

[0121] Figure 6 A schematic diagram of a process for determining the distance between locations provided in an embodiment of the present application;

[0122] Figure 7 A schematic diagram of a process for determining size differences provided in an embodiment of the present application;

[0123] Figure 8 Another schematic diagram of a target matching method provided in an embodiment of the present application;

[0124] Fig. 9 A schematic diagram of a structure of a target matching device provided in an embodiment of the present application;

[0125] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0126] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0127] In order to meet the increasingly complex monitoring scenarios, the use of gun camera and ball camera linkage (referred to as gun-ball linkage) solutions is becoming more and more popular. The gun-ball linkage solution usually includes a gun camera and at least one ball camera, in which the gun camera is responsible for detecting the target, and then, according to the position of the target in the image obtained by the gun camera, the ball camera is controlled to move in the PTZ coordinate system, so that the ball camera can clearly capture the target detected by the gun camera, and then the ball camera is used to follow the target detected by the gun camera, determine the details, and other operations. It can be seen that in the gun-ball linkage solution, the target detected by the gun camera must be determined in the field of view of the ball camera, and then the subsequent steps can be implemented. However, after the ball camera moves in the PTZ coordinate system according to the position of the target in the gun camera, there may be multiple targets in the field of view of the ball camera. At this time, if it is impossible to accurately determine which specific target in the field of view of the ball camera is the target detected by the gun camera, the subsequent steps cannot be accurately executed.

[0128] For example, if the ball camera is required to follow the target detected by the gun camera, when the ball camera moves in the PTZ coordinate system according to the position of the target of interest in the gun camera, there are multiple targets in the field of view of the ball camera. At this time, if a target in the field of view of the ball camera (hereinafter referred to as the misidentified target) is mistakenly identified as the target detected by the gun camera, the subsequent following result of the misidentified target is obviously inaccurate, and if other judgments are made based on the following result, the judgment result will also be inaccurate. Based on this, how to accurately determine the target detected by the gun camera from the field of view of the ball camera is crucial.

[0129] For another example, in the alarm detection scenario of gun-ball linkage, the gun camera detects whether there is a target that meets the alarm conditions. When the gun camera detects the existence of a target that meets the alarm conditions, the PTZ coordinates that can make the target within the field of view of the ball camera are calculated. After that, the ball camera is controlled to move to the determined PTZ coordinates. Furthermore, the target detected by the gun camera is reviewed by the ball camera to determine whether the target detected by the gun camera really meets the alarm conditions. If so, it indicates that the target detected by the gun camera is a positive alarm. If not, it indicates that the target detected by the gun camera is a false alarm and the false alarm needs to be filtered.

[0130] However, in this process, after the ball camera moves to the determined PTZ coordinates, there may be multiple targets in the field of view, and it is impossible to determine which specific target is the target detected by the gun camera that meets the alarm condition, and thus, the ball camera cannot accurately review the target. For this situation, usually as long as the category of any target in the field of view of the ball camera is the target category specified in the alarm condition, it is considered that the target detected by the gun camera actually meets the alarm condition. For example, if the alarm condition is a person located in the alarm area, then the target category specified in the alarm condition is a person. At this time, there is an animal located in the alarm area and a person located outside the alarm area in the scene monitored by the ball grabbing linkage device. During the detection process, the gun camera mistakenly identifies the animal in the alarm area as a person in the alarm area. Afterwards, the PTZ coordinates that can make the mistaken person located in the field of view of the ball camera are calculated, and the ball camera is controlled to move to the PTZ coordinates. Assuming that at this time, the field of view of the ball camera not only includes the mistaken person, but also includes the aforementioned person outside the alarm area, then it is determined that the target detected by the gun camera actually meets the alarm condition. However, the actual situation is that the target detected by the gun camera is an animal rather than a person, that is, the target detected by the gun camera does not actually meet the alarm condition. It can be seen that in the alarm detection scenario of the gun-ball linkage, since it is impossible to accurately determine which specific target in the field of view of the ball camera is the target detected by the gun camera, it may lead to the inability to effectively filter out false alarms.

[0131] In the related art, in order to determine the target detected by the gun camera from multiple targets in the field of view of the ball camera, grayscale and contour similarity are usually used as the basis for judgment. However, for the gun-ball linkage scenario, using grayscale and contour similarity as the basis for judgment is not accurate. That is, using grayscale and contour similarity as the basis cannot accurately determine which specific target in the field of view of the ball camera is the target detected by the gun camera, that is, it is impossible to accurately determine whether two targets are the same target.

[0132] In order to more clearly illustrate that it is impossible to accurately determine whether two targets are the same target based on grayscale and contour similarity in the gun-ball linkage scenario, two specific scenarios in the alarm detection scenario of the gun-ball linkage are used as examples for explanation. It can be understood that the following example is only a possible application scenario provided by the present application where the above technical problem exists. In other possible embodiments, the above technical problem may also exist in other possible application scenarios, and the following example does not impose any limitation on this.

[0133] Scenario 1: For gun-ball linkage equipment, the thermal imaging lens in the gun camera is used to shoot thermal imaging images at a long distance. When a target that meets the alarm conditions is detected through the thermal imaging image, the PTZ coordinates that can make the target within the field of view of the ball camera are determined according to the position of the target. Then, the ball camera is controlled to move to the PTZ coordinates. Then, the target detected by the gun camera is reviewed by the ball camera to determine whether the target detected by the gun camera really meets the alarm conditions.

[0134] Scenario 2: For gun-ball linkage equipment, the visible light camera in the gun camera is used to shoot visible light images at a long distance. When a target that meets the alarm conditions is detected through the visible light image, the PTZ coordinates that can make the target within the field of view of the ball camera are determined according to the position of the target. Then, the ball camera is controlled to move to the PTZ coordinates. Furthermore, the target detected by the gun camera is reviewed by the ball camera to determine whether the target detected by the gun camera really meets the alarm conditions.

[0135] Regarding the above scenario one, in the related art, when there are multiple targets in the field of view of the ball camera, matching is performed through grayscale and contour similarity to determine whether there is a target in the field of view of the ball camera that matches the target detected by the gun camera, thereby improving the accuracy and effectiveness of false alarm filtering; however, for thermal imaging images and visible light images, the grayscale similarity between the target in the thermal imaging image and the target in the visible light image is not large (that is, regardless of whether it is the same target, the grayscale difference in the thermal imaging image and the visible light image is large, and it is not accurate to determine whether it is the same target based on the grayscale difference), and the thermal imaging image has the problem of unclear contours (that is, the contours of the objects in the thermal imaging image are unclear, therefore, regardless of whether it is the same target, the contours of the targets in the thermal imaging image and the visible light image are large, and it is not accurate to determine whether it is the same target based on the contour difference), therefore, it is not accurate to determine whether there is a target in the field of view of the ball camera that matches the target detected by the gun camera from the two dimensions of grayscale and contour.

[0136] For the second scenario mentioned above, although the image taken by the gun camera is a visible light image, due to the limitation of the magnification, the target in the visible light image taken by it will also have unclear outlines, and it is difficult to accurately determine the target in the field of view of the ball camera that matches the target detected by the gun camera based on the similarity of the outline.

[0137] Therefore, in view of the problem mentioned above that the target detected by the gun camera cannot be accurately determined from the field of view of the ball camera, the present application provides a target matching method for accurately determining the target detected by the gun camera from the field of view of the ball camera. The target matching method provided by the present application is applied to an electronic device, which may be a desktop computer, a portable computer, a smart mobile terminal, a gun camera in a gun-ball linkage device, a ball camera in a gun-ball linkage device, etc.

[0138] See also Figure 1 , Figure 1 A schematic diagram of the structure of a gun-ball linkage device provided in an embodiment of the present application, wherein the gun camera 1 includes a thermal imaging lens 11 and a visible light lens 12; the ball camera 2 includes a visible light lens 21. In a gun-ball linkage scenario, when the gun camera includes both a thermal imaging lens and a visible light lens, target detection can be performed based on an image captured by the thermal imaging lens in the gun camera, or based on an image captured by the visible light lens in the gun camera.

[0139] The target matching method provided in the embodiment of the present application is described in detail below through specific examples.

[0140] See also Figure 2 , Figure 2 A schematic flow chart of a target matching method provided in an embodiment of the present application includes the following steps:

[0141] Step S21: in response to the gun camera detecting the first target, calculating the PTZ coordinates that can make the position of the first target within the field of view of the ball camera, controlling the ball camera to move to the determined PTZ coordinates, and controlling the ball camera to remain stationary after moving to the determined PTZ coordinates;

[0142] Step S22: after the ball camera moves to the determined PTZ coordinates, the gun camera is controlled to shoot to obtain a plurality of first image frames, and the ball camera is controlled to synchronously shoot in a stationary state to obtain a plurality of second image frames, and a timestamp of each of the first image frames and each of the second image frames is obtained;

[0143] Step S23: Detect and obtain the second targets in each second image frame, and determine whether there is a third target that meets a preset matching condition in each second target.

[0144] The preset matching conditions include: the similarity between the second motion trace corresponding to the second target and the first motion trace is greater than the first preset similarity; the first motion trace is the motion trace of the first target determined according to each first image frame and the timestamp of each first image frame; the second motion trace is the motion trace of the second target determined according to each second image frame and the timestamp of each second image frame;

[0145] In the technical solution provided by the embodiment of the present application, in response to the gun camera detecting the first target, the PTZ coordinates that can make the position of the first target within the field of view of the ball camera are calculated, and then, after the ball camera is controlled to move to the determined PTZ coordinates, the first target is located within the field of view of the ball camera, and the ball camera can shoot the first target; the gun camera is controlled to shoot to obtain multiple first image frames, and the ball camera is controlled to synchronously shoot in a static state to obtain multiple second image frames, and the timestamps of each first image frame and each second image frame are obtained; then, the second target existing in each second image frame is detected, and it is determined whether there is a third target that meets the preset matching condition in each second target. Among them, the preset matching condition includes that the similarity between the second motion track corresponding to the second target and the first motion track is greater than the first preset similarity; and the first motion track is the motion track of the first target determined according to each first image frame and the timestamp of each first image frame; the second motion track is the motion track of the second target determined according to each second image frame and the timestamp of each second image frame.

[0146] It can be understood that for the gun-ball linkage scene, usually, the gun camera and the ball camera are shooting at a distance from the target, so the gun camera and the ball camera can be approximately regarded as being located at the same position. Therefore, for the same target, in the same time period, the shapes of its motion traces in the first image frame (i.e., the first motion trace) and the motion traces in the second image frame (i.e., the second motion trace) should be similar. Therefore, if the second motion trace corresponding to the second target is more similar to the first motion trace than the first preset similarity, it indicates that the second target is the third target, that is, the target that matches the first target detected by the gun camera. In this way, the target detected by the gun camera can be accurately determined from the field of view of the ball camera.

[0147] In the above step S21, the gun camera can shoot the first image frame in the following step S22 through the thermal imaging lens, or can shoot the first image frame in the following step S22 through the visible light lens. At the same time, it can be understood that when the gun camera shoots the first image frame through the thermal imaging lens, the gun camera can only have a thermal imaging lens, or can also have both a thermal imaging lens and a visible light lens; when the gun camera shoots the first image frame through the visible light lens, the gun camera can only have a visible light lens, or can also have both a visible light lens and a thermal imaging lens. The lens in the ball camera is a visible light lens, and the ball camera shoots the second image frame in the following step S22 through the visible light lens.

[0148] The gun is pre-configured with a target for detection, which may be a person, an animal, etc. During the operation of the gun, if the pre-configured target for detection is detected, the detected target is the first target. For example, the gun is pre-configured to detect a person, and during the operation of the gun, if a person is detected, the person is the first target.

[0149] PTZ coordinates can be recorded as (pan, tilt, zoom). PTZ coordinates can be used to achieve all-round (left / right / up / down) movement of sports equipment (such as a ball camera), as well as magnification and zoom control. Specifically, pan is used to control the left and right rotation of the ball camera, that is, to control the horizontal movement of the ball camera; tilt is used to control the up and down rotation of the ball camera, that is, to control the vertical movement of the ball camera; zoom is used to control the magnification and zoom function of the ball camera, that is, to control the ball camera to be able to pull in or out of the shooting target.

[0150] In the embodiment of the present application, in response to the gun camera detecting the first target, the electronic device can calculate the PTZ coordinates that can make the first target within the field of view of the ball camera according to the position of the first target detected by the gun camera, and control the ball camera to move to the PTZ coordinates, so that the first target exists in the field of view of the ball camera, and the image frame with the first target can be captured. It can be understood that, after the ball camera moves to the PTZ coordinates, it remains stationary.

[0151] In the above step S22, when the lens in the gun camera is a thermal imaging lens, the first image frame is a thermal imaging image; when the lens in the gun camera is a visible light lens, the first image frame is a visible light image. The lens in the ball camera is a visible light lens, so the second image frame is a visible light image. The timestamp of the first image frame is used to indicate the time when the gun camera took the first image frame; the timestamp of the second image frame is used to indicate the time when the ball camera took the second image frame.

[0152] In the embodiment of the present application, the first image frame is an image frame obtained by the gun camera shooting the first target after the ball camera moves to the determined PTZ coordinates; the second image frame is an image frame obtained by the ball camera synchronously shooting in a stationary state after the ball camera moves to the determined PTZ coordinates. Here, synchronization can be understood as that, for multiple first image frames shot by the gun camera and multiple second image frames shot by the ball camera, each first image frame has a second image frame with the same timestamp, and the number of first image frames is the same as the number of second image frames.

[0153] Specifically, in some embodiments, the same shooting frequency and the required number of frames to be shot can be set in advance for the ball camera and the gun camera. After the ball camera moves to the determined PTZ coordinates, the gun camera and the ball camera are controlled to start shooting at the same time. In this way, the gun camera shoots the first first image frame while the ball camera shoots the first second image frame; the gun camera shoots the second first image frame while the ball camera shoots the second second image frame. The principle of other first image frames and second image frames in sequence is the same. Therefore, for each first image frame, there is a second image frame with the same timestamp, and the number of first image frames is the same as the number of second image frames.

[0154] Alternatively, in some other embodiments, the same shooting frequency and the required number of frames to be shot may not be set for the ball camera and the gun camera, but a shooting duration may be set for the ball camera. In this case, when the gun camera detects a first target that meets the preset alarm condition, it can continuously shoot image frames for the first target. After the ball camera moves to the determined PTZ coordinate, the ball camera continuously shoots image frames within the set shooting time period. After the shooting duration reaches the set shooting duration, the electronic device obtains all image frames shot by the gun camera from the detection of the first target to the current moment (hereinafter referred to as the seventh image frame), and all image frames continuously shot by the ball camera within the set shooting duration (hereinafter referred to as the eighth image frame), and determines the image frames with the same timestamp from the seventh image frame and the eighth image frame, and uses the seventh image frame among the image frames with the same timestamp as the first image frame, and the eighth image frame as the second image frame. In this way, each first image frame has a second image frame with the same timestamp, and the number of first image frames is the same as the number of second image frames. In order to obtain a sufficient number of first image frames and second image frames for subsequent analysis (such as analysis required for subsequent determination of the similarity of motion traces and motion directions), the set shooting duration can be set to be longer.

[0155] It is understandable that the two methods of acquiring the first image frame and the second image frame provided here are only for illustrative purposes and do not serve as limitations. That is, in this step, as long as there is a second image frame with the same timestamp for each acquired first image frame, and the number of first image frames is the same as the number of second image frames, the specific acquisition steps are not limited.

[0156] In addition, the same timestamp in this article refers to the same within the allowable error range. For example, by default, the image in the field of view of a camera (such as a gun camera or a ball camera) will not change within 0.1ms, so the first image frame and the second image frame with a timestamp interval of 0.1ms can also be regarded as image frames with the same timestamp.

[0157] In the above step S23, the second target is the target obtained after the electronic device detects the second image frame. Specifically, when the electronic device determines the second target through the second image frame, it can detect any second image frame and use the detected target as the second target, or it can detect all the second image frames and use the detected targets as the second target. Alternatively, in order to reduce the amount of calculation and ensure accuracy, the second image frame with the earliest shooting time among multiple second image frames can be detected, and the detected target can be used as the second target.

[0158] It should be noted that in the embodiment of the present application, the category of the second target should be the same as the category of the first target, that is, the category of the target that the ball camera is configured to detect should be the same as the category of the target that the gun camera is configured to detect; for example, if the gun camera is configured to detect people, the ball camera should also be configured to detect people.

[0159] The first preset similarity can be set by the user according to actual conditions. For example, if the preset matching condition only includes that the second motion trace corresponding to the second target has a similarity with the first motion trace that is greater than the first preset similarity, the first preset similarity can be set higher to accurately determine the third target corresponding to the first target; if the preset matching condition also includes other conditions, the first preset similarity can be set not too high to enable the determination of the second target that satisfies multiple preset matching conditions at the same time. In the embodiment of the present application, if there are multiple second motion traces whose similarity with the first motion trace is greater than the first preset similarity, then in order to improve accuracy, the second target to which the trace with the greatest similarity to the first motion trace belongs among these second motion traces can be used as the third target.

[0160] The first motion trace is the motion trace of the first target determined by the electronic device according to each first image frame and the timestamp of each first image frame. The process of determining the first motion trace is described with an example. For example, according to each first image frame, the electronic device can determine the position of the first target in each first image frame as the position of the first target corresponding to each first image frame (hereinafter referred to as the third position), and then, the third position corresponding to each first image frame is sorted according to the timestamp of each first image frame to obtain the first motion trace.

[0161] More specifically, for example, the first image frames acquired by the electronic device are image frame 1, image frame 2, and image frame 3, the timestamp of image frame 1 is 10:01, the timestamp of image frame 2 is 10:02, and the timestamp of image frame 3 is 10:03; and the electronic device determines that the position of the first target in image frame 1 is position 1, the position in image frame 2 is position 2, and the position in image frame 3 is position 3, respectively. At this time, position 1, position 2, and position 3 can all be regarded as the third position. When position 1, position 2, and position 3 are sorted according to the timestamps of image frame 1, image frame 2, and image frame 3, position 1 can be used as the first position of the first motion trace, position 2 can be used as the second position of the first motion trace, and position 3 can be used as the third position of the first motion trace, so as to obtain the first motion trace. It can be understood that in this example, for the sake of ease of description, only three image frames are used as the first image frame for illustration, which does not serve as a limitation.

[0162] At the same time, based on the above content, it can be known that the various track points constituting the first motion track are the positions of the first target in each first image frame. Therefore, the first motion track can actually be understood as the motion track of the first target in the first image frame. Therefore, the first motion track in the embodiment of the present application is actually the motion track in the first coordinate system, wherein the first coordinate system is the image coordinate system of the first image frame.

[0163] After the electronic device acquires the second image frame, there may be multiple targets in the second image frame, and the electronic device cannot determine which specific target is the first target photographed by the gun. Therefore, for the acquired second image frame, the electronic device can use deep learning networks such as yoloV3 and SSD to detect the target of interest in the second image frame, and then use the detected targets as second targets to obtain multiple second targets, and construct a second motion track for each second target, so as to facilitate the subsequent analysis of which second target is the first target detected by the gun through the motion track.

[0164] The second motion trace is the motion trace of each second target determined by the electronic device according to the second image frame and the timestamp of each second image frame. The principle of determining each second motion trace is the same as the principle of determining the first motion trace described above. It is only necessary to replace the image frame in the example describing the first motion trace with the second image frame, and replace the first target with the second target to which the second motion trace belongs, which will not be repeated here.

[0165] And, similarly, each track point constituting the second motion track is the position of the second target in the second image frame. Therefore, the second motion track can actually be understood as the motion track of the second target to which the second motion track belongs in the second image frame. Therefore, the second motion track in the embodiment of the present application is actually the motion track in the second coordinate system, wherein the second coordinate system is the image coordinate system of the second image frame.

[0166] It can be understood that the electronic device determines the track points constituting the motion track based on the acquired image frames, but there may be a situation where the first target or a second target cannot be detected in some image frames. For this situation, the electronic device can use a position prediction algorithm to predict the position of the first target or the second target in these image frames according to the position of the first target or the second target in other image frames. For example, the multiple second image frames acquired by the electronic device are image frame 1, image frame 2, image frame 3, image frame 4, and image frame 5. The second target determined according to the second image frame includes target 1, but the electronic device only detects target 1 in image frame 1, image frame 2, image frame 3, and image frame 5, and does not detect image 1 in image frame 4. Then, a position prediction algorithm such as a convolutional neural network (CNN) can be used to predict the position of target 1 in image frame 4 according to the position of target 1 in image frame 1, image frame 2, image frame 3, and image frame 5. In this way, the number of track points constituting each motion track is the same, and the situation where the motion track of the target is greatly different due to missed detection and other reasons is reduced. Furthermore, in the embodiments of the present application, the electronic device may adopt a target following algorithm such as Hungarian to determine the motion track of each target, and, in order to distinguish different motion tracks, a unique identifier may be set for the motion track corresponding to each target.

[0167] See also Figure 3 , Figure 3 A schematic diagram of a flow chart for determining the similarity of motion traces provided in an embodiment of the present application may include the following steps:

[0168] Step S31: determining the minimum bounding rectangle of the first motion trace as the first bounding rectangle; and determining the minimum bounding rectangle of each second motion trace as the second bounding rectangle;

[0169] Step S32: scaling each track point in the first motion track according to the first circumscribed rectangle to obtain a third motion track; and scaling each track point in each second motion track according to each second circumscribed rectangle to obtain a fourth motion track corresponding to each second motion track;

[0170] wherein each fourth motion trace has the same size as the third motion trace;

[0171] Step S33: calculating the degree of overlap between each fourth motion trace and the third motion trace as the degree of similarity between each second motion trace and the first motion trace.

[0172] In the technical solution provided in the embodiment of the present application, it can be understood that for the scene of gun-ball linkage, usually, the gun camera and the ball camera are performing long-distance shooting at a place far away from the target. Therefore, the gun camera and the ball camera can be approximately located at the same position. Therefore, for the same target, in the same time period, at the same scale, its motion trace in the first image frame and the motion trace in the second image frame should overlap. Therefore, in the embodiment of the present application, after scaling the first motion trace and the second motion trace to the same size, the overlap degree of the first motion trace and each second motion trace is calculated as the similarity degree between each second motion trace and the first motion trace, thereby improving the accuracy of the determined third target, that is, improving the accuracy of determining the target detected by the gun camera from the field of view of the ball camera.

[0173] In the above step S31, the electronic device determines the minimum bounding rectangle of the first motion trace and the minimum bounding rectangle of each second motion trace, and uses the minimum bounding rectangle of the first motion trace as the first bounding rectangle and the minimum bounding rectangle of the second motion trace as the second bounding rectangle.

[0174] For ease of understanding, Figure 4 Taking the example shown in the figure as an example, the positional relationship between the motion trace and its minimum circumscribed rectangle is explained. Figure 4 The black points (such as point 41) can be understood as the positions of the same target in different image frames (such as the position of the first target in each first image frame); Figure 4 The track 42 formed by the six black dots is the movement track of the target (such as the first target); Figure 4 The middle frame 43 is the image frame to which the motion trace belongs (wherein the image frame to which the first motion trace belongs is the first image frame, and the image frame to which the second motion trace belongs is the second image frame); Figure 4 The middle frame 44 is the minimum circumscribed rectangle of the motion track 42. Figure 4 The vertex coordinates of the upper left corner of the minimum circumscribed rectangle 42 are ( , ), with a width of , the height is .and, Figure 4 The horizontal axis of the coordinate system is recorded as the X-axis, the vertical axis is recorded as the Y-axis, and the origin of the coordinate system is recorded as 0.

[0175] In the above step S32, the third motion trace is the trace obtained by the electronic device after scaling each trace point in the first motion trace according to the first circumscribed rectangle; the fourth motion trace is the trace obtained after scaling each trace point in the second motion trace according to each second circumscribed rectangle, and the third motion trace has the same size as the fourth motion trace.

[0176] Specifically, the electronic device can determine the coordinates of the track points constituting the third motion track based on any vertex coordinate of the first circumscribed rectangle, the width and height of the first circumscribed rectangle, and the coordinates of each track point in the first motion track, that is, determine the third motion track. The method for determining the fourth motion track is the same as the method for determining the third motion track. It is only necessary to replace the first circumscribed rectangle with the second circumscribed rectangle, and determine the coordinates of each track point in the first motion track as the coordinates of each track point in the second motion track. It should be noted that the vertex coordinates of the circumscribed rectangle selected when determining the third motion track and the fourth motion track must be the vertex coordinates of the same position. For example, when determining the third motion track, the vertex coordinates of the upper right corner of the first circumscribed rectangle are selected, and when determining the fourth motion track, the vertex coordinates of the upper right corner of the second circumscribed rectangle must also be selected.

[0177] Below, a specific example is used to illustrate the method of determining the third motion trace and the fourth motion trace in the embodiment of the present application.

[0178] For example, after determining the first circumscribed rectangle and each second circumscribed rectangle, the electronic device determines the vertex of the upper left corner of the first circumscribed rectangle, and the width and height of the first circumscribed rectangle; and determines the vertex of the upper left corner of each second circumscribed rectangle, and the width and height of each second circumscribed rectangle. Then, the coordinates of the trace points constituting the third and the coordinates of the trace points constituting each fourth motion trace are determined according to the following formulas (1)-(2):

[0179] (1)

[0180] (2)

[0181] Among them, when ( , ) is the coordinate of the i-th track point in the first motion track, ( , ) is the coordinate of the vertex of the upper left corner of the first circumscribed rectangle, is the width of the first circumscribed rectangle, is the height of the first circumscribed rectangle, ( , ) is the coordinate of the i-th track point in the third motion track; when ( , ) is the coordinate of the i-th track point in the second motion track, ( , ) are the coordinates of the vertex of the upper left corner of the second circumscribed rectangle; is the width of the second circumscribed rectangle, When is the height of the second circumscribed rectangle, ( , ) is the coordinate of the i-th track point in the fourth motion track.

[0182] In the above step S33, after determining the third motion trace and each fourth motion trace, the electronic device may calculate the degree of overlap between each fourth motion trace and the third motion trace as the degree of similarity between the second motion trace and the first motion trace.

[0183] It can be understood that for two tracks, the higher the degree of overlap between the tracks, the more likely it is that they are the tracks of the same target. Therefore, after determining the third motion track and each second motion track, the electronic device can determine, among each second motion track, a track whose degree of overlap with the third motion track is greater than a first preset similarity degree.

[0184] In some embodiments, the difference value between the third motion trace and each fourth motion trace can also be used to represent the overlap between the third motion trace and each fourth motion trace. It can be understood that the greater the difference value between the third motion trace and each fourth motion trace, the smaller the overlap; the smaller the difference value between the third motion trace and each fourth motion trace, the greater the overlap. Therefore, when the overlap between the third motion trace and each fourth motion trace is represented by the difference value between the third motion trace and each fourth motion trace, in each fourth motion trace, the trace whose difference value with the third motion trace is less than the difference threshold is determined as the fifth motion trace, and the third target corresponding to the first target is determined from the second target to which the fifth motion trace belongs.

[0185] In some embodiments, the electronic device may determine the difference between the third motion trace and each fourth motion trace by using the following formula (3):

[0186] (3)

[0187] Wherein, Dist represents the difference between the third motion trace and the fourth motion trace, ( , ) is the coordinate of the ith track point in the third motion track, ( , ) is the coordinate of the ith track point in the fourth motion track, and N is the number of track points constituting the motion track.

[0188] It can be understood that the track point can be regarded as the center point of the target box, which is the target box where the target is detected by the electronic device using a deep learning algorithm.

[0189] In some embodiments, the preset matching conditions may also include: the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity, wherein the movement direction of the first target is the movement direction of the first target during the period from the first first image frame obtained by shooting the first target to the last first image frame; the movement direction of the second target is the movement direction of the second target belonging to the second motion track during the period from the first second image frame obtained by shooting the second target belonging to the second motion track to the last second image frame.

[0190] Specifically, in an embodiment of the present application, the electronic device can use a fitting algorithm to fit the various track points constituting the first motion track to obtain the motion direction of the first target; and use a fitting algorithm to fit the various track points constituting the second motion track to obtain the motion direction of the second target.

[0191] Alternatively, in some embodiments, for the first target, the electronic device may determine the coordinates of the first target in the first first image frame obtained by photographing the first target as the first coordinates; and determine the coordinates of the first target in the last first image frame obtained by photographing the first target as the second coordinates; and then, calculate the direction of the straight line formed by the first coordinate and the second coordinate based on the first coordinate and the second coordinate as the movement direction of the first target.

[0192] For each second target, the electronic device can determine the coordinates of the second target in the first second image frame obtained by photographing the second target as the third coordinate; and determine the coordinates of the second target in the last second image frame obtained by photographing the second target as the fourth coordinate; and then, calculate the direction of the straight line formed by the third coordinate and the fourth coordinate based on the third coordinate and the fourth coordinate as the movement direction of the second target.

[0193] In an embodiment of the present application, when determining the movement direction of the second target, each second image frame is sorted from small to large according to the timestamp (that is, the time represented is from early to late). If the second target exists in the first second image frame, the first second image frame obtained by photographing the second target is the first second image frame after sorting; if the second target does not exist in the first second image frame after sorting, the coordinates of the second target in the first second image frame after sorting can be predicted based on the position of the second target in other second image frames, and then, the first second image frame obtained by photographing the second target is also the first first image frame after sorting; or, if the second target does not exist in the first second image frame after sorting, the first second image frame that actually photographs the second target can also be used as the first second image frame obtained by photographing the second target.

[0194] For example, the second target is target 1, and the second image frames are sorted from small to large by timestamp: image frame 1, image frame 2, image frame 3, image frame 4, and image frame 5. At this time, if target 1 exists in image frame 1, the coordinates of target 1 in image frame 1 are used as the third coordinates. If target 1 does not exist in image frame 1, the coordinates of target 1 in image frame 1 can be preset according to the positions of target 1 in other second image frames. Then, the coordinates of target 1 in image frame 1 are used as the third coordinates. Alternatively, if target 1 does not exist in image frame 1 and target 1 exists in image frame 2, the coordinates of target 1 in image frame 2 can also be used as the third coordinates.

[0195] It can be understood that the principle of determining the fourth coordinate is the same as the principle of determining the third coordinate, and it is only necessary to replace the first second image frame in the aforementioned content with the last second image frame.

[0196] Below, see Figure 5 Taking the determination of the moving direction of the first target as an example, the relationship between the first coordinate and the second coordinate and the moving direction of the first target is described again. It can be understood that the principle of determining the moving direction of the second target is the same as the principle of determining the moving direction of the first target. It is only necessary to change the following Figure 5 In the example shown, the first target is replaced by the second target, the first image frame is replaced by the second image frame, the first coordinate is replaced by the third coordinate, and the second coordinate is replaced by the fourth coordinate.

[0197] For example, the first target is target 1, Figure 5 The midpoint 51 is the coordinate of the target 1 in the first first image frame obtained by photographing the target 1 (i.e., the first coordinate); the point 52 is the coordinate of the target in the last first image frame obtained by photographing the target 1 (i.e., the second coordinate); Figure 5 The track 53 formed by the black dots is the movement track of the target 1; Figure 5 The middle dashed line 54 indicates the direction of movement of the target 1; Figure 5 The middle frame 55 is the first image frame. Figure 5 The horizontal axis of the coordinate system is recorded as the X-axis, the vertical axis is recorded as the Y-axis, and the origin of the coordinate system is recorded as 0.

[0198] Specifically, the electronic device can determine the moving direction of the first target according to the first coordinate and the second coordinate, and determine the moving direction of the second target according to the third coordinate and the fourth coordinate by the following formula (4):

[0199] (4)

[0200] Among them, atan2 is the representation of the inverse tangent function (i.e., arctan()) in the programming language; when ( , ) is the first coordinate, ( , ) is the second coordinate, is the moving direction of the first target; when ( , ) is the third coordinate, ( , ) is the fourth coordinate, is the movement direction of the second target to which the third coordinate and the fourth coordinate belong.

[0201] It can be understood that for two tracks, the more similar the directions of the tracks are, the more likely they are the tracks of the same target. Therefore, after determining the movement direction of the first target and the movement directions of each second target, the electronic device can determine, among the movement directions of each second target, the movement direction whose degree of similarity with the movement direction of the first target is greater than a second preset degree of similarity, and determine the second target to which the determined direction belongs as the third target.

[0202] In some embodiments, if the second preset similarity level is set to B, then ,or ,or , can be regarded as the similarity between the moving direction of the second target and the moving direction of the first target is greater than the second preset similarity. represents the moving direction of the first target, Indicates the moving direction of the second target.

[0203] In the embodiment of the present application, the second preset similarity can be set by the user according to actual needs. It can be understood that when the preset matching conditions include that the similarity between the first motion track corresponding to the second target and the first motion track is greater than the first preset similarity, and also include that the similarity between the motion direction of the second target and the motion direction of the first target is greater than the second preset similarity, only when the similarity between the motion track of a second target and the first motion track is greater than the first preset similarity, and the similarity between the motion direction of the second target and the motion direction of the first target is greater than the second preset similarity, the second target can be determined as the third target. Therefore, in order to avoid the situation where the third target cannot be found, it is not advisable to set the first preset similarity and the second preset similarity too large.

[0204] In addition, when the preset matching conditions include that the degree of similarity between the first motion track corresponding to the second target and the first motion track is greater than the first preset similarity, and also include that the degree of similarity between the motion direction of the second target and the motion direction of the first target is greater than the second preset similarity, if there is only one second target that satisfies both the degree of similarity between the motion track and the first preset similarity and the degree of similarity between the motion direction and the second preset similarity, then this second target is the third target; if there are multiple second targets that satisfy both the degree of similarity between the motion track and the first preset similarity and the degree of similarity between the motion directions and the second preset similarity, then which second target to use as the third target can be determined based on the degree of similarity between the motion tracks of these multiple second targets and the first target and the degree of similarity between the motion directions. For example, when the degree of similarity between the second motion track of a certain second target and the first motion track, and the degree of similarity between its motion direction and the motion direction of the first target are higher than those of other second targets, then this second target can be used as the third target.

[0205] If there is no second target that satisfies both the similarity of the motion trajectory being greater than the first preset similarity and the similarity of the motion direction being greater than the second preset similarity, it can be considered that there is no target matching the first target in the field of view of the ball camera.

[0206] In the technical solution provided in the embodiment of the present application, it can be understood that for the gun-ball linkage scene, usually, the gun camera and the ball camera are used for long-distance shooting at a distance from the target. Therefore, the gun camera and the ball camera can be approximately regarded as being located at the same position. Therefore, for the same target, within the same time period, the direction of its running track in the first image frame should be consistent with the direction of its running track in the second image frame. Therefore, in the embodiment of the present application, the third target corresponding to the first target is determined based on both the similarity of the motion track and the similarity of the motion direction, thereby improving the accuracy of determining the third target from the second target.

[0207] In some embodiments, the preset matching condition may further include: the distance between the position of the second target and the position of the first target is less than the preset distance. Figure 6 Taking the illustrated process as an example, the method for determining the distance between the position of the second target and the position of the first target provided in an embodiment of the present application is described.

[0208] See also Figure 6 , Figure 6 A schematic diagram of a flow chart for determining the distance between locations provided in an embodiment of the present application may include the following steps:

[0209] Step S61: selecting at least one image frame from the first image frame as the third image frame; and determining an image frame corresponding to the third image frame from the second image frame as the fourth image frame corresponding to the third image frame;

[0210] The fourth image frame corresponding to the third image frame is an image frame with the same timestamp as the third image frame.

[0211] Step S62: for each third image frame, projecting the position of the first target in the third image frame and the position of each second target in the fourth image frame corresponding to the third image frame into the target coordinate system;

[0212] Step S63: In the target coordinate system, calculate the distance between the position of each second target after projection and the position of the first target after projection, and obtain the distance between the position of each second target and the position of each first target. In the technical solution provided in the embodiment of the present application, at least one image frame is selected from the first image frame as the third image frame, and the image frame corresponding to the third image frame is determined from the second image frame as the fourth image frame corresponding to the third image frame, so that at least one group of third image frames and fourth image frames with the same timestamp are determined from the first image frame and the second image frame; then, for each third image frame, the position of the first target in the third image frame and the position of each second target in the fourth image frame corresponding to the third image frame are projected to the target coordinate system; then, in the target coordinate system, calculate the distance between the position of each second target after projection and the position of the first target after projection, and obtain the distance between the position of each second target and the position of each first target.

[0213] In the above step S61, the electronic device may select at least one image frame from the first image frame as the third image frame, and determine an image frame corresponding to the third image frame from the second image frame as a fourth image frame corresponding to the third image frame. The fourth image frame corresponding to the third image frame is an image frame with the same timestamp as the third image frame.

[0214] To illustrate with a specific example, for example, the first image frames acquired by the electronic device are image frame 1, image frame 2, image frame 3, image frame 4, and image frame 5; and the second image frames acquired by the electronic device are image frame 6, image frame 7, image frame 8, image frame 9, and image frame 10. Among them, image frame 1 has the same timestamp as image frame 6, image frame 2 has the same timestamp as image frame 7, image frame 3 has the same timestamp as image frame 8, image frame 4 has the same timestamp as image frame 9, and image frame 5 has the same timestamp as image frame 10. If the third image frame selected by the electronic device is image frame 1 and image frame 3, the electronic device will select image frame 6 as the fourth image frame corresponding to image frame 1, and select image frame 8 as the fourth image frame corresponding to image frame 3.

[0215] In the above step S62, for each third image frame, the electronic device may project the position of the first target in the third image frame to the target coordinate system, and project each second target in the fourth image frame corresponding to the third image frame to the target coordinate system. The target coordinate system may be any coordinate system. In some embodiments, for ease of calculation, the coordinate system of the first image frame or the coordinate system of the second image frame may be used as the target coordinate system. For details, please refer to the examples shown in the subsequent content.

[0216] Continuing with the example shown in the above step S61, when the third image frame is image frame 1, the fourth image frame corresponding to the third image frame is image frame 6, then the electronic device projects the position of the first target in image frame 1 to the target coordinate system, and projects the position of each second target in image frame 6 to the target coordinate system; at the same time, when the third image frame is image frame 3, the fourth image frame corresponding to the third image frame is image frame 8, then the electronic device projects the position of the first target in image frame 3 to the target coordinate system, and projects the position of each second target in image frame 8 to the target coordinate system.

[0217] In the above step S63, in the target coordinate system, the electronic device calculates the distance between the position of each second target after projection and the position of the first target after projection, and obtains the distance between the position of each second target and the position of the first target.

[0218] Specifically, when the electronic device only selects one third image frame, after projecting the position of the first target in the third image frame to the target coordinate system, and projecting the positions of each second target in the fourth image frame corresponding to the fourth image feature to the target coordinate system, the distance between the position of each second target after projection and the first position of the first target after projection is directly used as the distance between the position of each second target and the position of the first target.

[0219] For example, the third image frame is image frame 1, and the position of target 1 (i.e., the first target) in image frame 1 is position 1; the fourth image frame corresponding to the third image frame (i.e., the fourth image frame corresponding to image frame 1) is image frame 5, and the second targets in image frame 5 are target 2 and target 3, and the position of target 2 in image frame 5 is position 2, and the position of target 3 in image frame 5 is position 3. Position 4 is obtained by projecting position 1 into the target coordinate system, position 5 is obtained by projecting position 2 into the target coordinate system, and position 6 is obtained by projecting position 3 into the target coordinate system. Afterwards, in the target coordinate system, the distance between position 5 and position 4 is calculated as the distance between the position of target 2 and the position of target 1; and the distance between position 6 and position 4 is calculated as the distance between the position of target 3 and the position of target 1.

[0220] When the electronic device selects multiple third image frames, for each third image frame, the position of the first target in the third image frame is projected to the target coordinate system, and the position of each second target in the fourth image frame corresponding to the fourth image frame is projected to the target coordinate system. Thus, for each third image frame and the fourth image frame corresponding to the third image frame, the distance between the position of each second target after projection and the position of the first target after projection is determined, so that for each second target, there are multiple distances between the position of the second target and the position of the first target (here, when there are multiple third image frames, the distance between the position of the second target and the position of the first target calculated based on one first image frame and one fourth image frame is recorded as the first distance). In this case (i.e., there are multiple first distances for each second target), for each second target, the electronic device can calculate the average of the multiple first distances of the second target, and use the average as the distance between the position of the second target and the position of the first target; or, the distance with the largest value can be selected from the multiple first distances of the second target as the distance between the position of the second target and the position of the first target; or, the distance with the smallest value can be selected from the multiple first distances of the second target as the distance between the position of the second target and the position of the first target.

[0221] Below, the calculation of the mean is used as an example for explanation, for example, the third image frame is image frame 1 and image frame 2; the fourth image frame corresponding to image frame 1 is image frame 3, and the fourth image frame corresponding to image frame 2 is image frame 4. The position of the first target in image frame 1 is position 1, and the position of the first target in image frame 2 is position 2; the second target in image frame 3 is target 1 and target 2, the position of target 1 in image frame 3 is position 3, and the position of target 2 in image frame 3 is position 4; the second target in image frame 4 is target 1 and target 2, the position of target 1 in image frame 4 is position 5, and the position of target 2 in image frame 4 is position 6. The electronic device projects position 1, position 3, and position 4 to the target coordinate system, calculates the distance between the projected position 3 and the projected position 1 as distance 1, and calculates the distance between the projected position 4 and the projected position 1 as distance 2. Afterwards, the electronic device projects position 2, position 5, and position 6 to the target coordinate system, calculates the distance between the projected position 5 and the projected position 2 as distance 3, and calculates the distance between the projected position 6 and the projected position 2 as distance 4. In this way, distance 1 and distance 3 are both the distances between target 1 and the first target (i.e., the two first distances of target 1), and distance 2 and distance 4 are both the distances between target 2 and the first target (i.e., the two first distances of target 2). At this time, the average of distance 1 and distance 3 can be calculated as the distance between target 1 and the first target, and the average of distance 2 and distance 4 can be calculated as the distance between target 2 and the first target.

[0222] Next, a specific example is used to illustrate the above method of determining the distance between the position of each second target and the position of the first target. For example, taking the gun linkage device including a visible light camera and a thermal imaging camera, and the target coordinate system being the first coordinate system as an example, for example, a group of third image frames and fourth image frames with the same timestamp are determined, wherein the third image frame is image frame 1 and the fourth image frame is image frame 2, then the distance between the position of each second target and the position of the first target can be determined by the following (a)-(d):

[0223] (a) According to the three-dimensional positioning of the camera model, coordinate 1 of target 1 and coordinate 2 of target 2 in image frame 2 are mapped to the PTZ coordinate system of the ball camera to obtain coordinate 3 and coordinate 4; among which, target 1 and target 2 are the second targets.

[0224] (b) Using the homography matrix calibration relationship, coordinates 3 and 4 are mapped to the image coordinate system of the visible light image of the gun to obtain coordinates 5 and 6.

[0225] (c) Through dual-light fusion calibration, coordinates 5 and 6 are mapped to the image coordinate system of image frame 1 to obtain coordinates 7 and 8.

[0226] (d) Subtract coordinates 7 and 8 from coordinate 9 of the first target in image frame 1, and take the second motion trace of the second target whose coordinates have a difference value less than a first preset threshold from coordinate 9 as the sixth motion trace. For example, coordinate 7 is specifically (x F ,y F ), coordinate 9 is specifically (x L ,y L ), the first preset threshold is C, and ,and , then the second motion trajectory formed by the second target belonging to coordinate 7 is the sixth motion trajectory.

[0227] For this example, in some embodiments, the electronic device may also directly map the coordinate 1 of target 1 and the coordinate 2 of target 2 in image frame 2 to the coordinate system of image frame 1 according to the conversion relationship between the second coordinate system (i.e., the coordinate system of image frame 2) and the first coordinate system (i.e., the coordinate system of image frame 1); or, in some embodiments, the electronic device may first map the coordinate 1 of target 1 and the coordinate 2 of target 2 in image frame 2 to the PTZ coordinate system of the ball camera according to the conversion relationship between the second coordinate system and the PTZ coordinate system of the ball camera to obtain coordinate 3 and coordinate 4, and then map coordinate 3 and coordinate 4 to the first coordinate system (i.e., the coordinate system of image frame 1) according to the conversion relationship between the PTZ coordinate system and the first coordinate system. Here, the specific conversion process is not limited.

[0228] In the present application, when the preset matching conditions include that the degree of similarity between the first motion track corresponding to the first target and the first motion track is greater than the first preset similarity, and the distance between the position of the second target and the position of the first target is less than the preset distance, if there is only one second target that satisfies the conditions that the degree of similarity between the motion tracks is greater than the first preset similarity and the distance between the positions is less than the preset distance, then the second target is the third target; if there are multiple second targets that satisfy the conditions that the degree of similarity between the motion tracks is greater than the first preset similarity and the distance between the positions is less than the preset distance, then which second target to use as the third target can be determined based on the degree of similarity between the motion tracks of these multiple second targets and the distance between the positions. For example, when the second motion track of a second target is more similar to the first motion track than other second targets, and the distance between the position of the second target and the position of the first target is smaller, then the second target can be used as the third target.

[0229] If there is no second target that satisfies both the similarity degree of motion traces being greater than the first preset similarity degree and the distance between their positions being less than the preset distance, it can be considered that there is no target matching the first target in the field of view of the ball camera.

[0230] In the technical solution provided in the embodiment of the present application, it can be understood that for the gun-ball linkage scene, usually, the gun camera and the ball camera are used for long-distance shooting at a distance from the target. Therefore, the gun camera and the ball camera can be approximately regarded as being located at the same position. Therefore, for the same target, in the same time period, in the same coordinate system, its position in the first image frame and its position in the second image frame should be consistent. Therefore, in the embodiment of the present application, the third target corresponding to the first target is determined based on both the similarity of the motion trajectory and the distance between the positions, thereby improving the accuracy of determining the third target from the second target.

[0231] In some embodiments, the preset matching condition may further include: the size difference between the second object and the first object is less than the preset difference. Figure 7 Taking the illustrated process as an example, the method for determining the size difference between the size of the second target and the size of the first target provided in an embodiment of the present application is described.

[0232] See also Figure 7 , Figure 7 A schematic diagram of a process for determining a size difference provided in an embodiment of the present application may include the following steps:

[0233] Step S71: selecting a preset number of image frames from the first image frame as the fifth image frame, and selecting a preset number of image frames from the second image frame as the sixth image frame;

[0234] Step S72: for the fifth image frame and the sixth image frame, determining the size of the first object in the fifth image frame, and determining the size of each second object in the sixth image frame;

[0235] Step S73: Calculate the difference between the size of the first object and the size of each second object to obtain the size difference between the size of each second object and the size of the first object.

[0236] In the technical solution provided in the embodiment of the present application, a preset number of image frames are selected from the first image frame as the fifth image frame, and a preset number of image frames are determined from the second image frame as the sixth image frame; thereafter, for each fifth image frame and the sixth image frame, the size of the first target in each fifth image frame is determined, and the size of each second target in each sixth image frame is determined, and then, the difference value between the size of the first target and the size of each second target is calculated to obtain the size difference between the size of each second target and the size of the first target.

[0237] In the above step S71, the electronic device may select a preset number of image frames from the first image frame as the fifth image frame, and determine a preset number of image frames from the second image frame as the sixth image frame. The determined sixth image frame may be an image frame with the same timestamp as the fifth image frame, or may be an image frame with a different timestamp from the fifth image frame. The preset number may be set by the user according to actual needs. For example, if you want to improve accuracy, you may set the preset number to be larger. If you want to reduce the amount of calculation, you may set the preset number to be smaller.

[0238] To illustrate with a specific example, for example, the first image frames acquired by the electronic device are image frame 1, image frame 2, image frame 3, image frame 4, and image frame 5; and the second image frames acquired by the electronic device are image frame 6, image frame 7, image frame 8, image frame 9, and image frame 10; the preset number is 2. At this time, the electronic device selects image frame 1 and image frame 2 from the first image frame as the fifth image frame, and the electronic device can select any two image frames from the second image frame as the sixth image frame, or the electronic device can also select image frame 6 and image frame 7 with the same timestamp as image frame 1 and image frame 2 as the sixth image frame.

[0239] In the above step S72, for the fifth image frame and the sixth image frame, the electronic device may determine the size of each first object in the fifth image frame, and determine the size of each second object in the sixth image frame.

[0240] Regarding the size of the first target, in some embodiments, the electronic device may map the four vertex coordinates of the first target frame in the fifth image frame to the PTZ coordinate system according to the coordinate conversion relationship between the coordinate system of the fifth image frame and the PTZ coordinate system, obtain the projected vertex coordinates, and determine the size of the first target according to the width and height of the area enclosed by the projected vertex coordinates; wherein the first target frame is the target frame in which the first target is located when the first target in the fifth image frame is identified. Alternatively, in other embodiments, the electronic device may determine the size of each first target according to the field of view of the gun, the width and height of the area occupied by the first target in the fifth image frame, and the width and height of the fifth image frame.

[0241] Regarding the size of the second target, in some embodiments, the electronic device can map the four vertex coordinates of each second target frame in the sixth image frame to the PTZ coordinate system according to the coordinate conversion relationship between the coordinate system of the sixth image frame and the PTZ coordinate system, obtain the projected vertex coordinates, and determine the size of each second target according to the width and height of the area surrounded by the projected vertex coordinates; wherein the second target frame is the target frame where the second target is located when the second target in the sixth image frame is identified. Alternatively, in other embodiments, the electronic device can determine the size of the second target according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame.

[0242] Specifically, based on the coordinate conversion relationship between the coordinate system of the first image frame and the PTZ coordinate system, the four vertex coordinates of the first target frame in the fifth image frame are mapped to the PTZ coordinate system to obtain the projected vertex coordinates, and the size of the first target is determined by the width and height of the area enclosed by the projected vertex coordinates. As an example, the method of calculating the size of the first target is explained.

[0243] For example, the electronic device can calculate the size of the first target using the following formulas (5)-(6).

[0244] pan_field L =(pan right_top +pan right_bottom -pan left_top -pan left_bottom ) / 2 (5)

[0245] tilt_field L =(tilt right_top +tilt left_top -tilt right_bottom -tilt left_bottom ) / 2 (6)

[0246] Among them, pan_field L The width of the first target; tilt_field L is the height of the first target; (pan right_top , tilt right_top ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper right corner of the first target frame; (pan right_bottom , tilt right_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the lower right corner of the first target frame; (pan left_top , pan left_bottom) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper left corner of the first target frame; (pan left_bottom , tilt left_bottom ) is the projected vertex coordinates obtained by projecting the coordinates of the vertex in the lower left corner of the first target box.

[0247] Taking the example of determining the size of the second target based on the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame, the method of calculating the size of the second target is described.

[0248] For example, the electronic device can be determined by the following formulas (7)-(8):

[0249] pan_field F =pan_field zoom ×h target / h FOV (7)

[0250] tilt_field F =tilt_field zoom × target / w FOV (8)

[0251] Among them, pan_field F The width of the second target; tilt_field F is the height of the second target; (pan_field zoom , tilt_field zoom ) is the field of view of the ball camera; (h target , w target ) is the width and height of the area occupied by the second target in the second image frame; (h FOV , w FOV ) are the width and height of the second image frame.

[0252] Based on this, after determining the size of the first target and the size of each second target, it is possible to determine whether the size of the first target and the size of the second target are less than a preset difference by setting an upper threshold and a lower threshold. For example, the lower threshold is set to D1 and the upper threshold is set to D2. When D1 < pan_field L / pan_field F <D2, or, D1 <tilt_field L / tilt_field L When <D2, it can be considered that the size of the first target and the size of the second target are smaller than the preset difference.

[0253] In step S73, the electronic device calculates the difference between the size of the first object and the sizes of each second object, and obtains the size difference between the size of each second object and the size of the first object.

[0254] Specifically, when the electronic device only selects a fifth image frame and a sixth image frame, the size of the first target in the fifth image frame can be calculated, and the size of each second target in the sixth image frame can be calculated. Thereafter, the size difference between the size of each second target and the size of the first target is directly used as the size difference between the size of each second target and the size of the first target.

[0255] For example, the fifth image frame is image frame 1, and the size of the first target in image frame 1 is size 1; the sixth image frame is image frame 5, and the second targets in image frame 5 are target 2 and target 3, and the size of target 2 is size 2 and the size of target 3 is size 3, then the size difference between size 2 and size 1 is calculated as the size difference between target 2 and target 1; and the size difference between size 3 and size 1 is calculated as the size difference between target 3 and target 1.

[0256] When the electronic device selects multiple fifth image frames, for each fifth image frame and sixth image frame, the electronic device can calculate the size of the first target in each fifth image frame, and calculate the size of each second target in each sixth image frame, and then calculate the mean of the sizes of each first target, and calculate the mean of the sizes of each second target based on the size of each second target in each sixth image frame, and then use the size difference between the mean of the size of each second target and the mean of the size of the first target as the size difference between the size of each second target and the size of the first target.

[0257] Alternatively, in some embodiments, if the selected sixth image frame is an image frame with the same timestamp as the fifth image frame, then for multiple groups of fifth image frames and sixth image frames with the same timestamp, the difference between the size of each second target in each group and the size of the first target can be calculated in groups, and for each second target, the difference between its size determined according to the fifth image frame and a sixth image frame is recorded as the first size difference, so that for multiple groups of fifth image frames and sixth image frames with the same timestamp, each second target has multiple first size differences. In this case, for each second target, the electronic device can calculate the average of the multiple first size differences of the second target, and use the average as the difference between the size of the second target and the size of the first target; or, the size difference with the largest value can be selected from the multiple first size differences of the second target as the difference between the size of the second target and the size of the first target; or, the size difference with the smallest value can be selected from the multiple first size differences of the second target as the difference between the size of the second target and the size of the first target.

[0258] Below, the calculation of the mean is taken as an example for explanation, for example, the fifth image frame is image frame 1 and image frame 2; the sixth image frame with the same timestamp as image frame 1 is image frame 3, and the sixth image frame with the same size as image frame 2 is image frame 4. The size of the first target in image frame 1 is size 1, and the size of the first target in image frame 2 is size 2; the second target in image frame 3 is target 1 and target 2, the size of target 1 in image frame 3 is size 3, and the size of target 2 in image frame 3 is size 4; the second target in image frame 4 is target 1 and target 2, the size of target 1 in image frame 4 is size 5, and the size of target 2 in image frame 4 is size 6. Afterwards, for image frame 1 and image frame 3, the electronic device can calculate the size difference between size 3 and size 1 as difference 1, and the size difference between size 4 and size 1 as difference 2; for image frame 2 and image frame 4, the electronic device can calculate the size difference between size 5 and size 2 as difference 3, and the size difference between size 6 and size 2 as difference 4. In this way, Difference 1 and Difference 3 are both the size differences between target 1 and the first target (i.e., the two first size differences of target 1), Difference 2 and Difference 4 are both the size differences between target 2 and the first target (i.e., the two first size differences of target 2). At this time, the mean of Difference 1 and Difference 3 can be calculated as the size difference between target 1 and the first target, and the mean of Difference 2 and Difference 4 can be calculated as the size difference between target 2 and the first target.

[0259] In the present application, when the preset matching conditions include that the degree of similarity between the first motion track corresponding to the first target and the first motion track is greater than the first preset similarity, and the size difference between the second target and the first target is less than the preset difference, if there is only one second target that satisfies the conditions that the degree of similarity between the motion tracks is greater than the first preset similarity and the size difference is less than the preset difference, then the second target is the third target; if there are multiple second targets that satisfy the conditions that the degree of similarity between the motion tracks is greater than the first preset similarity and the size difference is less than the preset difference, then which second target is to be used as the third target can be determined based on the degree of similarity and size difference between the motion tracks of these multiple second targets and the first target. For example, when the second motion track of a second target is more similar to the first motion track than other second targets, and the size difference between the second target and the first target is smaller, then the second target can be used as the third target.

[0260] If there is no second target that satisfies both the similarity of the motion trace being greater than the first preset similarity and the size difference being less than the preset difference, it can be considered that there is no target matching the first target in the field of view of the ball camera.

[0261] In the technical solution provided in the embodiment of the present application, the third target corresponding to the first target is determined based on both the similarity of the motion traces and the size difference, thereby improving the accuracy of determining the third target from the second target.

[0262] In some embodiments, the preset matching conditions may also include: the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset similarity, the distance between the position of the second target and the position of the first target is less than a preset distance, and the size difference between the size of the second target and the size of the first target is less than a preset difference.

[0263] Based on the above content, in the present application, the electronic device may determine the third target according to any one or more of the matching conditions shown in the following (a)-(d).

[0264] (a) the degree of similarity between the second motion track corresponding to the second target and the first motion track is greater than a first preset degree of similarity;

[0265] (b) the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity;

[0266] (c) the distance between the position of the second target and the position of the first target is less than a preset distance;

[0267] (d) A size difference between the size of the second object and the size of the first object is smaller than a preset difference.

[0268] In some embodiments, the first target in the above step S21 may be a potential target that meets the preset alarm conditions. At this time, if the electronic device determines that there is a third target, the first target is determined to be a target that meets the preset alarm conditions. If there is no third target, the first target is determined to be a target that does not meet the preset alarm conditions.

[0269] In the embodiment of the present application, when the gun camera detects a target that meets the alarm conditions and the ball camera successfully verifies it, it can be determined that the target detected by the gun camera is a target that truly meets the alarm conditions. Therefore, the first target detected by the gun camera that meets the alarm conditions but has not yet been verified by the ball camera is called a potential target that meets the preset alarm conditions.

[0270] Among them, the gun camera review means that after the ball camera moves to the determined PTZ coordinates, it is determined whether there is the first target detected by the gun camera in the field of view of the ball camera, that is, whether there is a third target (the specific determination method can refer to the relevant content of the aforementioned preset matching conditions). When it is determined that there is a third target, it is considered that the ball camera review is successful, and the first target detected by the gun camera is determined to be a target that meets the preset alarm conditions; when there is no third target, it is considered that the ball camera review has failed, and the first target detected by the gun camera is determined to be a target that does not meet the preset alarm conditions.

[0271] The preset alarm conditions are conditions for triggering an alarm set by the user according to actual needs, that is, when the gun camera detects a potential target that meets the preset alarm conditions and the ball camera verifies successfully, the alarm is triggered; specifically, the preset alarm conditions in the embodiments of the present application can be the presence of a person, the presence of a person maintaining a specific posture, the presence of a person located in the alarm area, the presence of an animal located in the alarm area, etc.

[0272] It can be understood that when the first target in the above step S21 is a potential target that meets the preset alarm condition, the category that the gun camera is configured to detect is the category indicated by the preset alarm condition, and the category of the target that the ball camera is configured to detect should be the same as the category of the target that the gun camera is configured to detect. Therefore, at this time, the category that the ball camera is configured to detect in the above step S23 should also be the category indicated by the preset alarm condition. Specifically, for example, when the preset alarm condition is the presence of a person in the alarm area, the category indicated in the preset alarm condition is a person, then the gun camera should be configured to detect a person, and the ball camera should also be configured to detect a person; or, when the preset alarm condition is the presence of an animal in the alarm area, the category indicated in the preset alarm condition is an animal, then the gun camera should be configured to detect an animal, and the ball camera should also be configured to detect an animal.

[0273] Next, an example is used to illustrate the target matching method provided in the embodiment of the present application. Figure 8 , Figure 8 Another flowchart of the target matching method provided in the embodiment of the present application may include the following steps:

[0274] Step S81: After the thermal imaging gun camera image detects the target frame, the coordinates of the target frame to be verified are sent to the visible light ball camera in real time.

[0275] The target frame in the embodiment of the present application can be understood as the first target mentioned above, and details can be found in the description of step S21 above.

[0276] Step S82: After the ball camera obtains the first frame of the target of the gun camera, it will enlarge the center of the target and then keep it still for N frames.

[0277] The first frame target of the gun camera in the embodiment of the present application can be understood as the first target in the above step S21. After the ball camera obtains the first target, it determines the PTZ coordinates that can make the position of the first target within the field of view of the ball camera. After the ball camera moves to the determined PTZ coordinates, it controls the gun camera to shoot to obtain multiple first image frames, and controls the ball camera to synchronously shoot in a static state to obtain multiple second image frames. Here, N frames can be understood as the number of second image frames. For details, please refer to the description in the above step S22.

[0278] Step S83: Calculate the similarity of the movement track, movement direction, position and size of the target of the gun camera and the ball camera in the same time period when the ball camera is stationary for N frames. Figure 3 , Figure 6 , Figure 7 , and related descriptions for determining the degree of similarity of movement directions.

[0279] Step S84: If the movement tracks, movement directions, positions and sizes of the ball camera and the gun camera are similar, it is considered that the visible light target of the ball camera and the thermal imaging target of the gun camera are the same target, and the thermal imaging gun camera alarm target is credible.

[0280] In the embodiment of the present application, the movement track, movement direction, position, and size of the ball camera and the gun camera can be understood as the movement track, movement direction, position, and size of the target in the ball camera and the gun camera, as well as the position and size of the target in the ball camera and the gun camera. If the movement track, movement direction, position, and size of a certain target in the ball camera are similar to the movement track, movement direction, position, and size of the target detected by the gun camera (i.e., the target that meets the preset alarm conditions) (i.e., the aforementioned four preset matching conditions are met at the same time), then it is considered that the target in the ball camera (i.e., the aforementioned third target) is the target detected by the gun camera, that is, the target detected by the gun camera exists in the field of view of the ball camera, and the target detected by the gun camera is the target that truly meets the preset alarm conditions.

[0281] Corresponding to the above target matching method, the embodiment of the present application also provides a target matching device, such as Fig. 9 As shown, the device comprises:

[0282] The motion control module 91 is used for calculating the PTZ coordinates that can make the position of the first target within the field of view of the ball camera in response to the gun camera detecting the first target, and controlling the ball camera to move to the determined PTZ coordinates, and controlling the ball camera to remain stationary after moving to the determined PTZ coordinates;

[0283] The image acquisition module 92 is used to control the gun camera to shoot a plurality of first image frames after the ball camera moves to the determined PTZ coordinates, and to control the ball camera to synchronously shoot a plurality of second image frames in a stationary state, and obtain the timestamps of each first image frame and each second image frame;

[0284] The target determination module 93 is used to detect the second target existing in each second image frame, and determine whether there is a third target that meets the preset matching conditions in each second target; wherein the preset matching conditions include: the degree of similarity between the second motion trace corresponding to the second target and the first motion trace is greater than the first preset similarity; the first motion trace is the motion trace of the first target determined according to each first image frame and the timestamp of each first image frame; the second motion trace is the motion trace of the second target determined according to each second image frame and the timestamp of each second image frame.

[0285] In the technical solution provided by the embodiment of the present application, in response to the gun camera detecting the first target, the PTZ coordinates that can make the position of the first target within the field of view of the ball camera are calculated, and then, after the ball camera is controlled to move to the determined PTZ coordinates, the first target is located within the field of view of the ball camera, and the ball camera can shoot the first target; the gun camera is controlled to shoot to obtain multiple first image frames, and the ball camera is controlled to synchronously shoot in a static state to obtain multiple second image frames, and the timestamps of each first image frame and each second image frame are obtained; then, the second target existing in each second image frame is detected, and it is determined whether there is a third target that meets the preset matching condition in each second target. Among them, the preset matching condition includes that the similarity between the second motion track corresponding to the second target and the first motion track is greater than the first preset similarity; and the first motion track is the motion track of the first target determined according to each first image frame and the timestamp of each first image frame; the second motion track is the motion track of the second target determined according to each second image frame and the timestamp of each second image frame.

[0286] It can be understood that for the gun-ball linkage scene, usually, the gun camera and the ball camera are shooting at a distance from the target, so the gun camera and the ball camera can be approximately regarded as being located at the same position. Therefore, for the same target, in the same time period, the shapes of its motion traces in the first image frame (i.e., the first motion trace) and the motion traces in the second image frame (i.e., the second motion trace) should be similar. Therefore, if the second motion trace corresponding to the second target is more similar to the first motion trace than the first preset similarity, it indicates that the second target is the third target, that is, the target that matches the first target detected by the gun camera. In this way, the target detected by the gun camera can be accurately determined from the field of view of the ball camera.

[0287] In some embodiments, the first target is a potential target that meets a preset alarm condition, and the device further includes:

[0288] The alarm determination module is used to determine that the first target is a target that meets the preset alarm condition if the third target exists; if the third target does not exist, determine that the first target is a target that does not meet the preset alarm condition.

[0289] In some embodiments, the similarity between each second motion trace and the first motion trace is determined in the following manner:

[0290] Determine the minimum circumscribed rectangle of the first motion trace as the first circumscribed rectangle; and respectively determine the minimum circumscribed rectangle of each second motion trace as the second circumscribed rectangle;

[0291] According to the first circumscribed rectangle, each track point in the first motion track is scaled to obtain a third motion track; and according to each second circumscribed rectangle, each track point in each second motion track is scaled to obtain a fourth motion track corresponding to each second motion track; wherein each fourth motion track has the same size as the third motion track;

[0292] The similarity between each fourth motion trace and the third motion trace is calculated as the similarity between each second motion trace and the first motion trace.

[0293] In some embodiments, each track point in the first motion track is scaled according to the first circumscribed rectangle to obtain a third motion track; and each track point in each second motion track is scaled according to each second circumscribed rectangle to obtain a fourth motion track corresponding to each second motion track, including:

[0294] Determine the coordinates of the vertex of the upper left corner of the first circumscribed rectangle, and the width and height of the first circumscribed rectangle; and respectively determine the coordinates of the vertex of the upper left corner of each second circumscribed rectangle, and the width and height of each second circumscribed rectangle;

[0295] The coordinates of the trace points constituting the third motion trace and the coordinates of the trace points constituting each fourth motion trace are determined by the following formula;

[0296]

[0297]

[0298] Among them, when ( , ) is the coordinate of the i-th track point in the first motion track, ( , ) is the coordinate of the upper left corner of the first circumscribed rectangle, is the width of the first circumscribed rectangle, is the height of the first circumscribed rectangle, ( , ) is the coordinate of the i-th track point in the third motion track; when ( , ) is the coordinate of the i-th track point in the second motion track, ( , ) are the coordinates of the upper left corner of the second circumscribed rectangle, is the width of the second circumscribed rectangle, When is the height of the second circumscribed rectangle, ( , ) is the coordinate of the i-th track point in the fourth motion track.

[0299] In some embodiments, the preset matching condition further includes that the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity;

[0300] The moving direction of the first target is the moving direction of the first target during a period from the first first image frame obtained by photographing the first target to the last first image frame;

[0301] The movement direction of each second target is the movement direction of the second target to which the second movement track belongs during a period from the first second image frame obtained by shooting the second target to the last second image frame.

[0302] In some embodiments, the direction of movement of the first target is determined by:

[0303] Determine the coordinates of the first target in the first first image frame obtained by photographing the first target as the first coordinates; and determine the coordinates of the first target in the last first image frame obtained by photographing the first target as the second coordinates;

[0304] Calculate the direction of the straight line formed by the first coordinate and the second coordinate according to the first coordinate and the second coordinate as the moving direction of the first target;

[0305] For each second target, the movement direction of the second target is determined by:

[0306] Determine the coordinates of the second target in the first second image frame obtained by photographing the second target as the third coordinates; and determine the coordinates of the second target in the last second image frame obtained by photographing the second target as the fourth coordinates;

[0307] The direction of the straight line formed by the third coordinate and the fourth coordinate is calculated according to the third coordinate and the fourth coordinate as the moving direction of the second target to which the second moving track belongs.

[0308] In some embodiments, the preset matching condition further includes that the distance between the position of the second target and the position of the first target is less than a preset distance;

[0309] The distance between the position of each second target and the position of the first target is determined by the following method:

[0310] Selecting at least one image frame from the first image frame as the third image frame; and determining an image frame corresponding to the third image frame from the second image frame as a fourth image frame corresponding to the third image frame, wherein the fourth image frame has the same timestamp as the third image frame;

[0311] For each third image frame, projecting the position of the first target in the third image frame and the position of each second target in the fourth image frame corresponding to the third image frame into a target coordinate system;

[0312] In the target coordinate system, the distance between the position of each second target after projection and the position of the first target after projection is calculated to obtain the distance between the position of each second target and the position of the first target.

[0313] In some embodiments, the preset matching condition further includes that a size difference between the size of the second object and the size of the first object is less than a preset difference;

[0314] The size of the first target and the size of each second target are determined by:

[0315] Selecting a preset number of image frames from the first image frame as the fifth image frame, and selecting a preset number of image frames from the second image frame as the sixth image frame;

[0316] For the fifth image frame and the sixth image frame, determining the size of the first object in the fifth image frame, and determining the size of each second object in the sixth image frame;

[0317] The difference between the size of the first object and the size of each second object is calculated to obtain the size difference between the size of each second object and the size of the first object.

[0318] In some embodiments, determining the size of the first object in the fifth image frame includes,

[0319] According to the coordinate conversion relationship between the coordinate system of the fifth image frame and the PTZ coordinate system, the four vertex coordinates of the first target frame in the fifth image frame are mapped to the PTZ coordinate system to obtain the projected vertex coordinates; wherein the first target frame is the target frame where the first target is located when the first target in the fifth image frame is identified;

[0320] Determine the size of the first target according to the width and height of the area enclosed by the projected vertex coordinates;

[0321] Determining the size of each second object in the sixth image frame includes,

[0322] The size of each second target is determined according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame.

[0323] In some embodiments, the size of the first target is determined according to the projected vertex coordinates by the following formula:

[0324] pan_field L =(pan right_top +pan right_bottom -pan left_top -pan left_bottom ) / 2

[0325] tilt_field L =(tilt right_top +tilt left_top -tilt right_bottom -tilt left_bottom ) / 2

[0326] Among them, pan_field L The width of the first target; tilt_field L is the height of the first target; (pan right_top , tilt right_top ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper right corner of the first target frame; (pan right_bottom , tilt right_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the lower right corner of the first target frame; (pan left_top , pan left_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper left corner of the first target frame; (pan left_bottom , tilt left_bottom ) is the projected vertex coordinates obtained by projecting the coordinates of the vertex at the lower left corner of the first target frame;

[0327] The size of each second target is determined by the following formula according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame:

[0328] pan_field F =pan_field zoom ×h target / h FOV

[0329] tilt_field F =tilt_field zoom × target / w FOV

[0330] Among them, pan_field F The width of the second target; tilt_field F is the height of the second target; (pan_field zoom , tilt_field zoom ) is the field of view of the ball camera; (h target , w target ) are the width and height of the area occupied by the second target in the sixth image frame; (h FOV , w FOV ) are the width and height of the sixth image frame.

[0331] In some embodiments, the preset matching conditions also include: the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity, the distance between the position of the second target and the position of the first target is less than a preset distance, and the size difference between the size of the second target and the size of the first target is less than a preset difference.

[0332] In some embodiments, the lens in the gun camera is a thermal imaging lens, and the lens in the ball camera is a visible light lens.

[0333] The present application also provides an electronic device, such as Fig.10 As shown, including:

[0334] Memory 101, used for storing computer programs;

[0335] The processor 102 is configured to implement any of the above-mentioned target matching methods when executing the program stored in the memory 101.

[0336] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 102, the communication interface, and the memory 101 communicate with each other via the communication bus.

[0337] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0338] The communication interface is used for communication between the above electronic device and other devices.

[0339] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0340] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0341] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the target matching methods described above are implemented.

[0342] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any target matching method described in the above embodiments.

[0343] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0344] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0345] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, storage medium, and computer program, which refer to product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0346] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A target matching method, characterized in that: The method comprises: In response to the gun camera detecting the first target, calculating PTZ coordinates that can make the position of the first target within the field of view of the ball camera, controlling the ball camera to move to the determined PTZ coordinates, and controlling the ball camera to remain stationary after moving to the determined PTZ coordinates; After the ball camera moves to the determined PTZ coordinates, the gun camera is controlled to shoot to obtain a plurality of first image frames, and the ball camera is controlled to synchronously shoot to obtain a plurality of second image frames in a stationary state, and a timestamp of each of the first image frames and each of the second image frames is obtained; Detect and obtain the second target existing in each of the second image frames, and determine whether there is a third target satisfying a preset matching condition in each of the second targets; wherein the preset matching condition includes: the degree of similarity between the second motion trace corresponding to the second target and the first motion trace is greater than a first preset degree of similarity; the first motion trace is the motion trace of the first target determined based on each of the first image frames and the timestamps of each of the first image frames; the second motion trace is the motion trace of the second target determined based on each of the second image frames and the timestamps of each of the second image frames.

2. The method according to claim 1, characterized in that The first target is a target that potentially meets a preset alarm condition, and the method further includes: If there is a third target, determining that the first target is a target that meets the preset alarm condition; If the third target does not exist, the first target is determined to be a target that does not meet the preset alarm condition.

3. The method according to claim 1, characterized in that The similarity between each of the second motion traces and the first motion trace is determined in the following manner: Determine the minimum circumscribed rectangle of the first motion trace as a first circumscribed rectangle; and respectively determine the minimum circumscribed rectangle of each of the second motion traces as a second circumscribed rectangle; According to the first circumscribed rectangle, each track point in the first motion track is scaled to obtain a third motion track; and according to each of the second circumscribed rectangles, each track point in each of the second motion tracks is scaled to obtain a fourth motion track corresponding to each of the second motion tracks; wherein each of the fourth motion tracks has the same size as the third motion track; The degree of overlap between each of the fourth motion traces and the third motion trace is calculated as the degree of similarity between each of the second motion traces and the first motion trace.

4. The method according to claim 3, characterized in that The method of scaling each track point in the first motion track according to the first circumscribed rectangle to obtain a third motion track; and scaling each track point in each second motion track according to each second circumscribed rectangle to obtain a fourth motion track corresponding to each second motion track, includes: Determine the coordinates of the vertex of the upper left corner of the first circumscribed rectangle, and the width and height of the first circumscribed rectangle; and respectively determine the coordinates of the vertex of the upper left corner of each of the second circumscribed rectangles, and the width and height of each of the second circumscribed rectangles; The coordinates of the trace points constituting the third motion trace and the coordinates of the trace points constituting each of the fourth motion traces are determined by the following formula; ; ; Among them, when ( , ) is the coordinate of the i-th track point in the first motion track, ( , ) is the coordinate of the vertex of the upper left corner of the first circumscribed rectangle, is the width of the first circumscribed rectangle, is the height of the first circumscribed rectangle, ( , ) is the coordinate of the i-th track point in the third motion track; when ( , ) is the coordinate of the i-th track point in the second motion track, ( , ) is the coordinate of the upper left corner of the second circumscribed rectangle, is the width of the second circumscribed rectangle, is the height of the second circumscribed rectangle, ( , ) are the coordinates of the i-th track point in the fourth motion track.

5. The method according to claim 1, characterized in that The preset matching condition also includes that the degree of similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset degree of similarity; The movement direction of the first target is the movement direction of the first target during a period from the first first image frame obtained by photographing the first target to the last first image frame; The movement direction of each second target is the movement direction of the second target to which the second movement track belongs during a period from a first second image frame obtained by photographing the second target to which the second movement track belongs to to a last second image frame.

6. The method according to claim 5, characterized in that The moving direction of the first target is determined by: Determine the coordinates of the first target in a first first image frame obtained by photographing the first target as the first coordinates; and determine the coordinates of the first target in a last first image frame obtained by photographing the first target as the second coordinates; Calculate the direction of the straight line formed by the first coordinate and the second coordinate according to the first coordinate and the second coordinate as the moving direction of the first target; For each of the second targets, the movement direction of the second target is determined by: Determine the coordinates of the second target in the first second image frame obtained by photographing the second target as the third coordinates; and determine the coordinates of the second target in the last second image frame obtained by photographing the second target as the fourth coordinates; The direction of the straight line formed by the third coordinate and the fourth coordinate is calculated according to the third coordinate and the fourth coordinate as the moving direction of the second target to which the second moving track belongs.

7. The method according to claim 1, characterized in that The preset matching condition also includes that the distance between the position of the second target and the position of the first target is less than a preset distance; The distance between the position of each second target and the position of the first target is determined by: Selecting at least one image frame from the first image frame as a third image frame; and determining an image frame corresponding to the third image frame from the second image frame as a fourth image frame corresponding to the third image frame, wherein the fourth image frame corresponding to the third image frame is an image frame with the same timestamp as the third image frame; For each of the third image frames, projecting the position of the first target in the third image frame and the position of each of the second targets in the fourth image frame corresponding to the third image frame into a target coordinate system; In the target coordinate system, the distance between the position of each second target after projection and the position of the first target after projection is calculated to obtain the distance between the position of each second target and the position of the first target.

8. The method according to claim 1, characterized in that The preset matching condition also includes that the size difference between the size of the second object and the size of the first object is less than a preset difference; The size of the first target and the size of each of the second targets are determined by: Selecting a preset number of image frames from the first image frames as fifth image frames, and selecting the preset number of image frames from the second image frames as sixth image frames; For the fifth image frame and the sixth image frame, determine the size of the first object in the fifth image frame, and determine the size of each of the second objects in the sixth image frame; The difference between the size of the first object and the size of each of the second objects is calculated to obtain the size difference between the size of each of the second objects and the size of the first object.

9. The method according to claim 8, characterized in that The determining the size of the first object in the fifth image frame includes: According to the coordinate conversion relationship between the coordinate system of the fifth image frame and the PTZ coordinate system, the four vertex coordinates of the first target frame in the fifth image frame are mapped to the PTZ coordinate system to obtain the projected vertex coordinates; wherein the first target frame is the target frame where the first target is located when the first target in the fifth image frame is identified; Determine the size of the first object according to the width and height of the area enclosed by the projected vertex coordinates; The determining the size of each of the second objects in the sixth image frame includes: The size of each second target is determined according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame.

10. The method according to claim 9, characterized in that The size of the first target is determined according to the projected vertex coordinates by the following formula: pan_field L =(pan right_top +pan right_bottom -pan left_top -pan left_bottom ) / 2; tilt_field L =(tilt right_top +tilt left_top -tilt right_bottom -tilt left_bottom ) / 2; Among them, pan_field L The width of the first target; tilt_field L is the height of the first target; (pan right_top , tilt right_top ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper right corner of the first target frame; (pan right_bottom , tilt right_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the lower right corner of the first target frame; (pan left_top , pan left_bottom ) is the projected vertex coordinates obtained by projecting the vertex coordinates of the upper left corner of the first target frame; (pan left_bottom , tilt left_bottom ) is the projected vertex coordinates obtained by projecting the coordinates of the vertex at the lower left corner of the first target frame; The size of each second target is determined by the following formula according to the field of view of the ball camera, the width and height of the area occupied by each second target in the sixth image frame, and the width and height of the sixth image frame: pan_field F =pan_field zoom ×h target / h FOV ; tilt_field F =tilt_field zoom ×w target / w FOV ; Among them, pan_field F The width of the second target; tilt_field F is the height of the second target; (pan_field zoom , tilt_field zoom ) is the field of view of the ball camera; (h target , w target ) is the width and height of the area occupied by the second target in the sixth image frame; (h FOV , w FOV ) are the width and height of the sixth image frame.

11. The method according to claim 1, characterized in that: The preset matching conditions also include: the similarity between the movement direction of the second target and the movement direction of the first target is greater than a second preset similarity, the distance between the position of the second target and the position of the first target is less than a preset distance, and the size difference between the size of the second target and the size of the first target is less than a preset difference.

12. The method according to claim 1, characterized in that The lens in the gun camera is a thermal imaging lens, and the lens in the ball camera is a visible light lens.

13. A target matching device, characterized in that: The device comprises: A motion control module, configured to calculate PTZ coordinates that enable the position of the first target to be located within the field of view of the ball camera in response to the gun camera detecting the first target, control the ball camera to move to the determined PTZ coordinates, and control the ball camera to remain stationary after moving to the determined PTZ coordinates; An image acquisition module, used for controlling the gun camera to shoot a plurality of first image frames after the ball camera moves to the determined PTZ coordinates, and controlling the ball camera to synchronously shoot a plurality of second image frames in a stationary state, and obtaining a timestamp of each of the first image frames and each of the second image frames; A target determination module is used to detect the second target existing in each of the second image frames, and determine whether there is a third target that meets the preset matching conditions in each of the second targets; wherein the preset matching conditions include: the degree of similarity between the second motion trace corresponding to the second target and the first motion trace is greater than a first preset degree of similarity; the first motion trace is the motion trace of the first target determined based on each of the first image frames and the timestamps of each of the first image frames; the second motion trace is the motion trace of the second target determined based on each of the second image frames and the timestamps of each of the second image frames.

14. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-12 when executing a program stored in a memory.

Images