A target tracking method, device and device for target tracking
By combining environmental feature encoding and visual features of the target entity, target encoding is generated, and based on similarity matching, the problem of target matching errors in multi-objective tracking is solved, improving tracking accuracy.
Patent Information
- Application Number
- CN202110732651.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-06-29
AI Technical Summary
In multi-target tracking, when there are many objects in the video screen or when the objects are close to each other, matching based on visual features and motion features can easily lead to target matching errors, affecting the accuracy of multi-target tracking results.
By encoding the target entity environmental features, combining visual features, generating target encodings, and matching them according to the similarity between target encodings, we can improve target recognition and tracking accuracy.
It effectively avoids matching errors caused by similar characteristics of the target entity, and improves the accuracy and distinction of multi-objective tracking.
Smart Images

Figure CN113344978B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a target tracking method, a device and a device for target tracking. Background Art
[0002] Multi-target tracking refers to simultaneously tracking multiple targets in a video screen in a continuous video sequence to obtain the motion trajectories of multiple targets.
[0003] At present, multi-target tracking is usually performed based on the visual features or motion features of objects in the video screen. However, when there are a large number of objects in the video screen, there may be multiple objects with similar visual features; or when two objects are close to each other, the two objects may have similar motion features. In the above cases, multi-target tracking based on visual features and motion features may lead to target matching errors, affecting the accuracy of multi-target tracking results. Summary of the invention
[0004] The embodiments of the present invention provide a target tracking method, a device and a device for target tracking, which can improve the accuracy of multi-target tracking results.
[0005] In order to solve the above problem, an embodiment of the present invention discloses a target tracking method, which includes:
[0006] Performing target detection on each image frame in the continuous image frames to determine a target entity of a preset entity category in each image frame and a region coordinate of each target entity;
[0007] Determine surrounding entities within a preset range of the region coordinates of each target entity;
[0008] According to the entity category and entity position of each surrounding entity, encode the dimension of each target entity in each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category;
[0009] Determine the target code of each target entity according to the environmental element code corresponding to each preset entity category of each target entity;
[0010] According to the similarity between the target code of each target entity in the previous frame and the target code of each target entity in the next frame, the tracking result of the target entity in the previous frame in the next frame is determined.
[0011] Optionally, determining surrounding entities within a preset range of the region coordinates of each target entity includes:
[0012] In each image frame, determine the center point of the region coordinates of each target entity;
[0013] Entities located on both sides of the center point and having a lateral distance less than a preset distance are determined as peripheral entities, and the preset distance is determined according to the area width of the current target entity.
[0014] Optionally, encoding the dimensions of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity to obtain the environmental element coding of each target entity corresponding to each preset entity category includes:
[0015] For each target entity, the target entity is encoded in the dimension of each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category;
[0016] The encoding of the target entity in the dimension of each preset entity category includes:
[0017] Dividing a pixel area at a preset distance on both sides of the center point of the target entity into a preset number of pixel blocks to obtain pixel blocks corresponding to the target entity;
[0018] For each preset entity category, identify whether each pixel block corresponding to the target entity contains a pixel point of the target entity of the preset entity category, and set a flag bit of each pixel block according to the identification result; obtain an environmental element code of the target entity corresponding to the preset entity category according to the flag bit of each pixel block corresponding to the target entity;
[0019] The preset entity categories are traversed to obtain the environmental element codes of each target entity corresponding to each preset entity category.
[0020] Optionally, setting a flag bit of each pixel block according to the recognition result includes:
[0021] Setting a flag bit of a first pixel block corresponding to the target entity to a first flag bit, wherein the first pixel block is a pixel block including pixel points of the target entity of the preset entity category;
[0022] The flag bit of the second pixel block corresponding to the target entity is set as the second flag bit, the second pixel block being a pixel block that does not include pixel points of the target entity of the preset entity category.
[0023] Optionally, determining the target code of each target entity according to the environmental element code of each preset entity category corresponding to each target entity includes:
[0024] The environmental element codes of each target entity corresponding to each preset entity category are concatenated to obtain the target code; or,
[0025] The environment element codes of each target entity corresponding to each preset entity category are concatenated to obtain the environment code vector of each target entity;
[0026] Perform vector encoding according to the visual features of each target entity to obtain a visual encoding vector of each target entity;
[0027] The environment encoding vector and visual encoding vector of each target entity are concatenated to obtain the target encoding of each target entity.
[0028] Optionally, the method further comprises:
[0029] In response to detecting that at least one target entity meets a tracking termination condition, tracking of the target entity is stopped.
[0030] Optionally, the method further comprises:
[0031] According to the tracking result of the target entity in the previous frame in the subsequent frame, the time region of each target entity in the continuous image frames is determined.
[0032] On the other hand, an embodiment of the present invention discloses a target tracking device, the device comprising:
[0033] A target detection module is used to perform target detection on each image frame in the continuous image frames, and determine the target entity of a preset entity category in each image frame and the regional coordinates of each target entity;
[0034] A surrounding determination module, used to determine surrounding entities within a preset range where the regional coordinates of each target entity are located;
[0035] An environmental coding module is used to encode the dimensions of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity, so as to obtain the environmental element coding of each target entity corresponding to each preset entity category;
[0036] A target coding module, used to determine the target coding of each target entity according to the environmental element coding of each target entity corresponding to each preset entity category;
[0037] The result determination module is used to determine the tracking result of the target entity in the previous frame in the next frame according to the similarity between the target code of each target entity in the previous frame and the target code of each target entity in the next frame.
[0038] Optionally, the surrounding determination module includes:
[0039] A center determination submodule, used to determine the center point of the region coordinates of each target entity in each image frame;
[0040] The entity determination submodule is used to determine that entities located on both sides of the center point whose lateral distance is less than a preset distance are peripheral entities, and the preset distance is determined according to the area width of the current target entity.
[0041] Optionally, the environment encoding module includes:
[0042] A traversal submodule is used to encode each target entity in the dimension of each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category;
[0043] Wherein, the traversal submodule includes:
[0044] A pixel division unit, used to divide a pixel area at a preset distance on both sides of the center point of the target entity into a preset number of pixel blocks to obtain pixel blocks corresponding to the target entity;
[0045] a flag setting unit, for identifying, for each preset entity category, whether each pixel block corresponding to the target entity contains a pixel point of the target entity of the preset entity category, and setting a flag bit of each pixel block according to the identification result; and obtaining an environmental element code of the target entity corresponding to the preset entity category according to the flag bit of each pixel block corresponding to the target entity;
[0046] The traversal unit is used to traverse the preset entity categories to obtain the environmental element codes corresponding to each preset entity category for each target entity.
[0047] Optionally, the flag setting unit includes:
[0048] A first setting subunit, configured to set a flag bit of a first pixel block corresponding to the target entity to a first flag bit, wherein the first pixel block is a pixel block including pixel points of the target entity of the preset entity category;
[0049] The second setting subunit is configured to set a flag bit of a second pixel block corresponding to the target entity to a second flag bit, wherein the second pixel block is a pixel block that does not include pixel points of the target entity of the preset entity category.
[0050] Optionally, the target encoding module includes:
[0051] The first encoding submodule is used to concatenate the environmental element codes corresponding to each preset entity category of each target entity to obtain a target code; or
[0052] A first concatenation submodule is used to concatenate the environment element codes corresponding to each preset entity category of each target entity to obtain an environment code vector of each target entity;
[0053] The second encoding submodule is used to perform vector encoding according to the visual features of each target entity to obtain a visual encoding vector of each target entity;
[0054] The second splicing submodule is used to splice the environment coding vector and the visual coding vector of each target entity to obtain the target coding of each target entity.
[0055] Optionally, the device further comprises:
[0056] The tracking termination module is used to stop tracking the target entity in response to detecting that at least one target entity meets the tracking termination condition.
[0057] Optionally, the device further comprises:
[0058] The time region determination module is used to determine the time region of each target entity in the continuous image frames according to the tracking result of the target entity in the previous frame in the next frame.
[0059] On the other hand, an embodiment of the present invention discloses a device for target tracking, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing any of the target tracking methods described above.
[0060] On the other hand, an embodiment of the present invention discloses a machine-readable medium having instructions stored thereon, which, when executed by one or more processors of a device, causes the device to execute one or more of the target tracking methods described above.
[0061] The embodiments of the present invention include the following advantages:
[0062] The embodiment of the present invention performs target detection on each image frame in the continuous image frames, determines the target entity of the preset entity category in each image frame and the regional coordinates of each target entity; and in each image frame, determines the surrounding entities within the preset range where the regional coordinates of each target entity are located. The entity category and entity position of each surrounding entity near the target entity are used to encode the dimension of the target entity in each preset entity category, and obtain the environmental element coding of the target entity corresponding to each preset entity category. The environmental element coding can reflect the environmental characteristics near the target entity, such as which entities of the preset entity category exist near the target entity. In a frame of image, there may be two or more entities with similar features, but the surrounding environments of these two or more entities with similar features may be different. The embodiment of the present invention introduces environmental element coding to the target entity, and reflects the surrounding environmental characteristics of the target entity in the target coding, which can increase the distinctiveness and recognition between the target entities.
[0063] Based on the environmental element coding of the target entity, the target coding of the target entity is determined, and according to the similarity between the target coding of each target entity in the previous frame and the target coding of each target entity in the next frame, the tracking result of the target entity in the previous frame in the next frame is determined. Since the target coding incorporates the information of the environmental element coding, the surrounding environmental features of the target entity can be reflected. The target coding is used to match the target entities of two adjacent frames. Compared with simple visual features or motion features, the distinctiveness and recognition of the target entity are increased, and the matching errors caused by the similarity of the features of multiple target entities can be avoided, thereby improving the accuracy of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0065] Figure 1 is a flowchart of steps of an embodiment of a target tracking method of the present invention;
[0066] Figure 2 It is a schematic diagram of the relationship between a target entity and a preset distance of the present invention;
[0067] Figure 3 Yes Figure 2 A schematic diagram showing a pixel area divided into a preset number of pixel blocks;
[0068] Figure 4 is a structural block diagram of an embodiment of a target tracking device of the present invention;
[0069] Figure 5 is a block diagram of an apparatus 800 for target tracking according to the present invention;
[0070] Figure 6 It is a schematic diagram of the structure of the server in some embodiments of the present invention. DETAILED DESCRIPTION
[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0072] Method Embodiment
[0073] Reference Figure 1 , shows a flowchart of a target tracking method embodiment of the present invention, which may specifically include the following steps:
[0074] Step 101, performing target detection on each image frame in the continuous image frames, determining a target entity of a preset entity category in each image frame and a region coordinate of each target entity;
[0075] Step 102, determining surrounding entities within a preset range where the regional coordinates of each target entity are located;
[0076] Step 103: Encode the dimensions of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity, and obtain the environmental element code of each target entity corresponding to each preset entity category;
[0077] Step 104, determining a target code for each target entity according to the environmental element code corresponding to each preset entity category of each target entity;
[0078] Step 105: Determine the tracking result of the target entity in the previous frame in the next frame according to the similarity between the target code of each target entity in the previous frame and the target code of each target entity in the next frame.
[0079] The target tracking method provided by the present invention can be used to track the target entity in continuous image frames. The target tracking method can be applied to electronic devices, including but not limited to: servers, smart phones, voice recorders, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc.
[0080] The continuous image frames may be image frames of a video captured in real time by a camera or a recording device, or may be image frames obtained in advance in other ways. The embodiment of the present invention does not limit the source and acquisition method of the continuous image frames. For example, the continuous image frames may come from video images directly captured by road cameras, surveillance cameras of unmanned supermarkets, etc., or may come from video images obtained after being pre-processed by a computer.
[0081] For any video segment, the segment is processed into frames to obtain continuous image frames. The embodiment of the present invention performs target detection on the continuous image frames frame by frame to determine the target entity of the preset entity category in each image frame and the region coordinates of each target entity.
[0082] In the embodiment of the present invention, the previous frame and the next frame in the continuous image frames refer to the previous frame and the next frame in two adjacent image frames in the continuous image frames.
[0083] The previous frame and the next frame of two adjacent image frames in the continuous image frames come from two frames with similar time in the video stream, for example, 100ms apart in time, so the displacement of each entity in the two adjacent image frames is relatively small. For the target entity to be tracked, the change of its surrounding environment is also relatively small. Therefore, the embodiment of the present invention encodes each target entity according to the surrounding entities near the target entity, and obtains the environmental element encoding of each preset entity category corresponding to each target entity. Based on the environmental element encoding of the target entity, the target encoding of the target entity is determined, and the tracking result of the target entity in the previous frame in the next frame is determined according to the similarity between the target encoding of each target entity in the previous frame and the target encoding of each target entity in the next frame. For example, the higher the similarity, the higher the probability that the two target entities are the same target entity. Since the target encoding incorporates the information of the environmental element encoding, the surrounding environment characteristics of the target entity can be reflected. The target encoding is used to match the target entities of the two adjacent frames. Compared with the simple visual features or motion features, the distinctiveness and recognition of the target entity are increased, and the matching error caused by the similarity of the characteristics of multiple target entities themselves can be avoided, and the accuracy of multi-target tracking can be improved.
[0084] The tracking result may include: whether the same target entity exists in the previous frame and the next frame in the continuous image frames, and the position relationship of the same target entity in the previous frame and the next frame.
[0085] For the continuous image frames to be tracked, the embodiment of the present invention first performs target detection on each image frame in the continuous image frames to identify each target entity of a preset entity category in each image frame and determine the region coordinates corresponding to each target entity.
[0086] The preset entity category may be a pre-specified entity category to be tracked. The embodiment of the present invention may track target entities of the preset entity category in continuous image frames. Through target detection, each target entity of the preset entity category in each image frame may be identified and the region coordinates corresponding to each target entity may be determined.
[0087] It should be noted that the target detection process can be implemented by a deep learning algorithm, for example, by a two-stage target detection algorithm such as a regional algorithm based on convolutional neural network features (R-CNN) and a fast regional algorithm based on convolutional neural network features (Faster R-CNN), or by a one-stage target detection algorithm such as a You Only Look Once (YOLO) and a Single Shot MultiBox Detector (SSD). The embodiment of the present invention does not limit the specific method of target detection.
[0088] For example, each frame in the continuous image frames can be input into a pre-trained target detection model in turn, and target detection can be performed on each frame through the target detection model to identify target entities of all preset entity categories in each frame and mark each identified target entity in each frame.
[0089] In an optional embodiment of the present invention, the preset entity categories include but are not limited to any one or more of the following: traffic lights, road traffic signs, vehicle movement signs, vehicles, pedestrians, and road-related entities. Among them, traffic lights are lights that direct traffic operations, generally consisting of red lights, green lights, and yellow lights (commonly known as traffic lights). According to categories, traffic lights can include: motor vehicle lights, non-motor vehicle lights, pedestrian crossing lights, direction indicators (arrow lights), lane lights, flashing warning lights, and road and railway level crossing lights. Road traffic signs are signs with graphic symbols for displaying traffic regulations and road information, such as displaying driving information such as place names, routes, directions, or displaying prohibitions, warnings, and other information. Vehicle movement signs are signs that indicate the movement of vehicles and pedestrians. Road-related entities may include related entities that may be set up on the road, such as street lights, billboards, light boxes, isolation belts, green belts, etc.
[0090] In an example, assuming that the preset entity categories include traffic lights and road traffic signs, by executing the target tracking method of the embodiment of the present invention, all traffic lights and all road traffic signs in continuous image frames can be tracked.
[0091] After performing target detection on a certain image frame in the continuous image frames, each target entity of all preset entity categories in the image frame and the region coordinates corresponding to each target entity can be obtained. The region coordinates include the vertex coordinates of a rectangular region that can contain the target entity image and has the smallest area.
[0092] Next, in each image frame, the surrounding entities within a preset range where the region coordinates of each target entity are located are determined.
[0093] In practical applications, if the distance between two entities in an image frame is far, it is generally believed that the two entities have a small correlation; if the distance is close, it is generally believed that the two entities have a large correlation. Therefore, in an embodiment of the present invention, for a target entity to be tracked, the surrounding entities within a preset range of the regional coordinates of the target entity are determined, and the surrounding entities are located near the target entity. In two adjacent image frames, the relative displacement of the target entity to be tracked and the surrounding entities near the target entity changes little. In an embodiment of the present invention, the entity category and entity position of each surrounding entity near the target entity are used to encode the dimension of the target entity in each preset entity category, and the environmental element code corresponding to each preset entity category of the target entity can be obtained. The environmental element code can reflect the environmental characteristics near the target entity, such as which entity categories of entities exist near the target entity, and which entity categories of entities do not exist near the target entity. In one frame of image, there may be two or more entities with similar features, but the surrounding environments of these two or more entities with similar features may be different. The embodiment of the present invention introduces environmental element coding to the target entity and reflects the surrounding environmental features of the target entity in the target coding, which can increase the distinctiveness and recognition between the target entities and improve the accuracy of the multi-target tracking results.
[0094] In an optional embodiment of the present invention, the step 102 of determining the surrounding entities within a preset range where the regional coordinates of each target entity are located includes:
[0095] Step S11, in each image frame, determining the center point of the region coordinates of each target entity;
[0096] Step S12: determine that entities located on both sides of the center point whose lateral distance is less than a preset distance are peripheral entities, and the preset distance is determined according to the area width of the current target entity.
[0097] In practical applications, since the positions of the target entities identified in two adjacent frames do not change much, there are also small differences in the lateral distribution of other entities near the same target entity. In addition, the closer the entity on the same horizontal coordinate is to the camera, the wider it is, and the size of the entities near it and the distance between entities will also increase proportionally, that is, the distance has little effect on the encoding of the target entity. Therefore, for a certain target entity, the embodiment of the present invention uses the horizontal coordinate as a reference and selects entities whose lateral distances on both sides of the center point of the regional coordinates of the target entity are less than a preset distance as peripheral entities.
[0098] In Example 1, it is assumed that the preset entity categories include traffic lights and road traffic signs. Target detection is performed on a certain image frame to identify the target entities of all traffic lights and road traffic signs in the image frame. For example, the following target entities are identified in the image frame: traffic light 1, traffic light 2, traffic light 3, road sign 1, road sign 2, and turn sign 1. Among them, traffic light 1, traffic light 2, and traffic light 3 are traffic lights, and road sign 1, road sign 2, and turn sign 1 are road traffic signs.
[0099] Taking the target entity traffic light 1 as an example, the center point of the regional coordinates of the target entity traffic light 1 is determined, and the entities located on both sides of the center point whose lateral distance is less than the preset distance are determined as peripheral entities. The center point can be calculated based on the vertex coordinates of the rectangular area of the target entity. It should be noted that the peripheral entities can be entities of all preset entity categories.
[0100] The preset distance can be determined according to the area width of the current target entity. For example, if the current target entity is traffic light 1, and the area width of the target entity traffic light 1 is m pixels, then the preset distance can be set to n units, that is, the preset distance is set to a width of n×m pixels.
[0101] The value of n may be an empirical value. In one example, n is 8. Of course, the value of n is not limited to 8. Figure 2 , which shows a schematic diagram of the relationship between a target entity and a preset distance of the present invention. Figure 2 As shown, L1 represents the target entity traffic light. Select the preset distance of n units on both sides of the center point of the target entity traffic light L1. Assume that n is 8, that is, select the preset distance of 8 units to the left of the center point of the target entity traffic light L1, and select the preset distance of 8 units to the right of the center point of the target entity traffic light L1. Among them, 8 units refers to the area width of 8 target entity traffic lights L1. In this example, Figure 2 The surrounding entities are determined within a preset distance of 8 units on both sides of the target entity traffic light L1. Figure 2 If the entity is within the lateral range shown in FIG. Figure 2 As shown, there are two surrounding entities within the preset distance of 8 units to the left of the target entity traffic light L1, namely surrounding entity T1 and surrounding entity L2. The entity type of T1 is a road traffic sign, and the entity type of L2 is a traffic light. There is a surrounding entity L3 within the preset distance of 8 units to the right of the target entity traffic light L1, and the entity type of L3 is a traffic light. It should be noted that Figure 2The eight rectangular boxes on both sides of the target entity L1 are used to represent the preset distance of eight units, and the height of the eight rectangular boxes is only used as a schematic display. In a specific implementation, when determining the preset range of the horizontal distance on both sides of the center point of the target entity, the vertical range of the horizontal distance is not limited. For example, the vertical range can be the entire vertical height of the current image frame.
[0102] In an optional embodiment of the present invention, step 103 encodes the dimension of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity to obtain the environmental element code of each target entity corresponding to each preset entity category, including:
[0103] For each target entity, the target entity is encoded in the dimension of each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category;
[0104] The encoding of the target entity in the dimension of each preset entity category includes:
[0105] Dividing a pixel area at a preset distance on both sides of the center point of the target entity into a preset number of pixel blocks to obtain pixel blocks corresponding to the target entity;
[0106] For each preset entity category, identify whether each pixel block corresponding to the target entity contains a pixel point of the target entity of the preset entity category, and set a flag bit of each pixel block according to the identification result; obtain an environmental element code of the target entity corresponding to the preset entity category according to the flag bit of each pixel block corresponding to the target entity;
[0107] The preset entity categories are traversed to obtain the environmental element codes of each target entity corresponding to each preset entity category.
[0108] The embodiment of the present invention traverses each target entity in each image frame, and encodes each target entity in the dimension of each preset entity category.
[0109] In Example 1, assuming that the preset entity categories include traffic lights and road traffic signs, the following target entities are identified in a certain image frame: traffic light 1, traffic light 2, traffic light 3, road sign 1, road sign 2, turn sign 1. Among them, traffic light 1, traffic light 2, and traffic light 3 are traffic lights, and road sign 1, road sign 2, and turn sign 1 are road traffic signs.
[0110] Assuming that the current target entity of the traversal is traffic light 1, the traffic light 1 is encoded in the dimension of each preset entity category to obtain the environmental element coding of the traffic light 1 corresponding to each preset entity category. First, the traffic light 1 is encoded in the dimension where the preset entity category is traffic signal light to obtain the environmental element coding of the traffic light 1 corresponding to the preset entity category of traffic signal light; then, the traffic light 1 is encoded in the dimension where the preset entity category is road traffic sign to obtain the environmental element coding of the traffic light 1 corresponding to the preset entity category of road traffic sign. In this way, the environmental element coding of the traffic light 1 corresponding to each preset entity category can be obtained.
[0111] In the same way, traffic light 2 can be encoded in the dimension of each preset entity category to obtain the environmental element code of each preset entity category corresponding to traffic light 2. And so on, until the above traffic light 1, traffic light 2, traffic light 3, road sign 1, road sign 2, turn sign 1 are encoded in the dimension of each preset entity category, the environmental element code of each target entity corresponding to each preset entity category can be obtained.
[0112] Taking the current target entity as traffic light 1 and the current preset entity category as traffic light as an example, the dimension of traffic light 1 in the preset entity category is encoded. Specifically, first divide the pixel area at a preset distance on both sides of the center point of the traffic light 1 area coordinates into a preset number of pixel blocks to obtain the pixel blocks corresponding to the current target entity. There is no restriction on the vertical height of the pixel block. The preset number can be an empirical value, for example, the preset number is 32. It should be noted that the value of the preset number of 32 is only used as an example. The larger the value of the preset number, the richer the information contained in the obtained environmental element code, but the amount of calculation will also increase. In practical applications, the value of the preset number can be flexibly set according to actual needs.
[0113] Reference Figure 3 , showing that Figure 2 Schematic diagram of dividing a pixel area into a preset number of pixel blocks. Figure 3 As shown, it can be Figure 2 The pixel area shown is divided into 32 pixel blocks, denoted as p1 to p32. The embodiment of the present invention sets the flag bit of each pixel block according to the recognition result of whether each pixel block corresponding to the current target entity contains the pixel point of the target entity of the current preset entity category.
[0114] Optionally, setting a flag bit of each pixel block according to the recognition result includes:
[0115] Setting a flag bit of a first pixel block corresponding to the target entity to a first flag bit, wherein the first pixel block is a pixel block including pixel points of the target entity of the preset entity category;
[0116] The flag bit of the second pixel block corresponding to the target entity is set as the second flag bit, the second pixel block being a pixel block that does not include pixel points of the target entity of the preset entity category.
[0117] Assume that the first flag bit is 1 and the second flag bit is 0. It is understood that the embodiment of the present invention does not limit the specific contents of the first flag bit and the second flag bit. Figure 3 In the example shown, it is assumed that the current target entity is traffic light 1 (L1) and the current preset entity category is traffic light. Figure 3 The flag of each pixel block is set to determine whether each of the 32 pixel blocks shown contains pixel points of the target entity whose entity category is a traffic light. For example, pixel block p1 does not contain pixel points of a traffic light, and the flag of pixel block p1 is set to 0. Pixel blocks p6, p7, and p8 contain pixel points of road traffic signs but do not contain pixel points of traffic lights, and the flags of pixel blocks p6, p7, and p8 are also set to 0. Pixel blocks p10 and p11 contain pixel points of traffic lights, so the flags of pixel blocks p10 and p11 can be set to 1. Similarly, the flags of pixel blocks p23, p24, and p25 are set to 1. It should be noted that pixel blocks p16 and p17 are pixel blocks corresponding to the current target entity traffic light 1. Since the current target entity traffic light 1 is also a target entity whose entity category is a traffic light, the flags of pixel blocks p16 and p17 can also be set to 1. Thus, the flag bit of each pixel block in the 32 pixel blocks corresponding to the current target entity traffic light 1 can be obtained, and these 32 flag bits are combined into the environmental element code of the current target entity traffic light 1 corresponding to the preset entity category of traffic light, such as [00000000011000011000001110000000].
[0118] In the same way, the current target entity Traffic Light 1 is encoded in the dimension of the preset entity category of the road traffic sign, and the environmental element code corresponding to the preset entity category of the road traffic sign of Traffic Light 1 is obtained. The environmental element code is also a 32-bit code, and each bit indicates whether the corresponding pixel block contains the pixel point of the target entity of the entity category of the road traffic sign.
[0119] After encoding the current target entity traffic light 1 in the dimensions of all preset entity categories, the environmental element codes of the traffic light 1 corresponding to each preset entity category can be obtained.
[0120] According to the method of encoding traffic light 1, traffic light 2 is taken as the current target entity, and traffic light 2 is encoded in the dimensions of all preset entity categories, so that the environmental element encoding of traffic light 2 corresponding to each preset entity category can be obtained. By analogy, the environmental element encoding of each preset entity category corresponding to each target entity in each image frame can be obtained. The environmental element encoding contains the encoding information of the entity category of the target entity itself, and also contains the encoding information of the entity category and entity position of the surrounding entities near the target entity. Therefore, the environmental element encoding contains richer information related to the target entity, which increases the distinctiveness and recognition of the target entity.
[0121] According to the environmental element codes of each target entity corresponding to each preset entity category, the target code of each target entity can be determined.
[0122] Furthermore, determining the target code of each target entity according to the environmental element code of each preset entity category corresponding to each target entity may include:
[0123] The environmental element codes of each target entity corresponding to each preset entity category are concatenated to obtain the target code.
[0124] In an embodiment of the present invention, the target code of the target entity can be directly determined based on the environmental element code. Taking Example 1 as an example, the preset entity categories include traffic lights and road traffic signs. For the target entity traffic light 1, the environmental element code corresponding to the preset entity category of the target entity traffic light 1 can be obtained, and the environmental element code corresponding to the preset entity category of the target entity traffic light 1 can be obtained. By splicing these two 32-bit environmental element codes, the target code of the target entity traffic light 1 can be obtained, and the target code is 64 bits.
[0125] It should be noted that the embodiment of the present invention does not limit the type and number of preset entity categories to be tracked. In the embodiment of the present invention, the length of the environmental element codes corresponding to different preset entity categories of the target entity is unified. For example, in the above example, the length of each environmental element code is 32 bits. Assuming that the length of each environmental element code is n and the number of preset entity categories is m, the length of each target code is m×n. For example, in Example 1, the length of each environmental element code is 32 bits, and the preset entity categories include traffic lights and road traffic signs, that is, the number of preset entity categories is 2, then the length of each target code is 2×32=64 bits. If the number of preset entity categories is 3, the length of each target code is 3×32=96 bits.
[0126] In order to further enrich the feature information contained in the target code, the embodiment of the present invention can also combine the environmental factor code of the target entity with the visual features of the target entity to obtain the target code of the target entity. The visual features include but are not limited to the image features of the target entity, such as color features, texture features, shape features, spatial relationship features, etc.
[0127] Furthermore, the determining the target code of each target entity according to the environmental factor code of each preset entity category corresponding to each target entity may also include:
[0128] The environment element codes of each target entity corresponding to each preset entity category are concatenated to obtain the environment code vector of each target entity;
[0129] Perform vector encoding according to the visual features of each target entity to obtain a visual encoding vector of each target entity;
[0130] The environment encoding vector and visual encoding vector of each target entity are concatenated to obtain the target encoding of each target entity.
[0131] In the embodiment of the present invention, each target entity is encoded in the dimension of each preset entity category. After obtaining the environmental element code corresponding to each preset entity category of each target entity, for each target entity, the environmental element code corresponding to each preset entity category can be spliced to obtain an environmental coding vector of length m×n, denoted as v1. v1 contains the environmental information near the target entity, but does not contain the visual information of the target entity. In addition, visual features can be extracted for each target entity, and vector encoding can be performed according to the visual features of each target entity to obtain the visual coding vector of each target entity, denoted as v2, assuming that the length of v2 is t. v2 contains the visual information of the target entity, but does not contain the environmental information near the target entity. In the embodiment of the present invention, the environmental coding vector v1 and the visual coding vector v2 of each target entity are spliced to obtain the target code of each target entity, denoted as v3, and the length of v3 is (m×n+t). v3 contains both the visual information of the target entity and the environmental information near the target entity. The target code v3 can more clearly express the comprehensive features of the surrounding environment and the details of the target entity, avoiding the problem of inaccurate matching caused by the similarity of a single feature.
[0132] In an optional embodiment of the present invention, determining the tracking result of the target entity in the previous frame in the next frame according to the similarity between the target code of each target entity in the previous frame and the target code of each target entity in the next frame includes:
[0133] Calculate the cosine similarity between the target encoding of each target entity in the previous frame and the target encoding of each target entity in the next frame;
[0134] If there is a first target entity in the previous frame, there is a second target entity in the next frame, and the cosine similarity between the target code of the first target entity and the target code of the second target entity is greater than a preset value, it is determined that the first target entity and the second target entity correspond to the same target entity.
[0135] After obtaining the target code of each target entity in each image frame, the cosine similarity between the target codes can be calculated for the target entities in two adjacent frames. In Example 2, it is assumed that three target entities are identified in the first frame of the two adjacent frames: traffic light 1, traffic light 2, and road sign 1. Three target entities are identified in the second frame: traffic light 1', traffic light 2', and road sign 2. For the target entity traffic light 1, the cosine similarity between the target code of traffic light 1 and the target code of traffic light 1' is calculated, the cosine similarity between the target code of traffic light 1 and the target code of traffic light 2' is calculated, and the cosine similarity between the target code of traffic light 1 and the target code of road sign 2 is calculated. Similarly, for the target entity traffic light 2, the cosine similarity between the target code of traffic light 2 and the target code of traffic light 1' is calculated, the cosine similarity between the target code of traffic light 2 and the target code of traffic light 2' is calculated, and the cosine similarity between the target code of traffic light 2 and the target code of road sign 2 is calculated. For the target entity road sign 1, the cosine similarity between the target code of road sign 1 and the target code of traffic light 1', the cosine similarity between the target code of road sign 1 and the target code of traffic light 2', and the cosine similarity between the target code of road sign 1 and the target code of road sign 2 are calculated respectively.
[0136] Determine whether there is a cosine similarity greater than a preset value among the 9 cosine similarities calculated above. Assuming that there is a first target entity (traffic light 1) in the previous frame and a second target entity (traffic light 1') in the next frame, and the cosine similarity between the target code of traffic light 1 and the target code of traffic light 1' is greater than a preset value, then it is determined that traffic light 1 and traffic light 1' match successfully, that is, it is determined that traffic light 1 and traffic light 1' correspond to the same target entity. Similarly, it is determined that traffic light 2 and traffic light 2' match successfully, that is, it is determined that traffic light 2 and traffic light 2' correspond to the same target entity.
[0137] In this example, target entity landmark 1 in the previous frame does not match any target entities in the next frame, indicating that target entity landmark 1 meets the tracking termination condition; target entity landmark 2 in the next frame does not match any target entities in the previous frame, indicating that target entity landmark 2 is a newly appeared target entity.
[0138] It should be noted that the embodiment of the present invention calculates the cosine similarity of the target codes of each target entity in two adjacent frames, and uses the cosine similarity of the target codes to match the target entities in two adjacent frames. In a specific implementation, calculating the similarity between the target codes of each target entity in two adjacent frames is not limited to using the cosine similarity method, and any existing method for calculating code similarity can be used.
[0139] In an optional embodiment of the present invention, the method may further include:
[0140] In response to detecting that at least one target entity meets a tracking termination condition, tracking of the target entity is stopped.
[0141] For a number of consecutive image frames, steps 101 to 105 are executed until at least one target entity is detected to meet the tracking termination condition, indicating that the tracking task of the target entity is completed and the tracking of the target entity can be stopped.
[0142] Furthermore, the tracking termination condition may include:
[0143] If there is a target entity in the previous frame that is not successfully matched with each target entity included in the next frame, it is determined that the target entity in the previous frame that is not successfully matched meets the tracking termination condition.
[0144] In Example 2, assume that three target entities are identified in the previous frame: traffic light 1, traffic light 2, and road sign 1. Three target entities are identified in the next frame: traffic light 1', traffic light 2', and road sign 2. After similarity calculation, the cosine similarity between the target entity road sign 1 in the previous frame and the three target entities in the next frame is less than the preset value, indicating that the road sign 1 in the previous frame has not been successfully matched with the target entities included in the next frame. It can be determined that road sign 1 meets the termination tracking condition and the tracking of road sign 1 can be stopped.
[0145] In an optional embodiment of the present invention, the method may further include:
[0146] If there is a target entity in the subsequent frame that is not successfully matched with any target entity included in the previous frame, the target entity in the subsequent frame that is not successfully matched is determined to be a new target entity.
[0147] In the above example 2, there is a target entity landmark 2 in the next frame that is not successfully matched with any target entity included in the previous frame, and the target entity landmark 2 that is not successfully matched in the next frame is determined to be a new target entity. That is, a new target entity appears in the next frame.
[0148] After a new target entity is detected, the target tracking method of the embodiment of the present invention can be used to track the existing target entity and the new target entity in subsequent continuous image frames. For example, in Example 2, after a new target entity road sign 2 is detected in a subsequent frame, the target entities traffic light 1', traffic light 2', and road sign 2 can be tracked in subsequent image frames of the subsequent frame.
[0149] In an optional embodiment of the present invention, the method may further include:
[0150] According to the tracking result of the target entity in the previous frame in the subsequent frame, the time region of each target entity in the continuous image frames is determined.
[0151] By executing the target tracking method of the embodiment of the present invention on the continuous image frames of a certain video, it is possible to identify all target entities of the preset entity categories appearing in the video and the time region of each target entity in the continuous image frames.
[0152] Specifically, based on the tracking results of the target entity in the previous frame in the subsequent frame of the continuous image frames, the starting frame and the ending frame of each target entity in the continuous image frames can be determined, and the time region of each target entity in the continuous image frames can be obtained based on the starting frame and the ending frame of each target entity.
[0153] The start frame refers to the first frame in which a target entity appears continuously in a number of frames, and the end frame refers to the last frame in which the target entity appears continuously in a number of frames. The start frame can be the first frame of the continuous image frames, or any frame other than the first frame. The number of frames can be one frame or more than one frame.
[0154] For example, for a certain video segment, the first frame includes a target entity, and the target entity is recorded as La 11 , then the first frame is the target entity La 11 Assume that the target entity La 11 After the first frame, it appears continuously until the fifth frame, and the target entity La does not appear in the sixth frame. 11 , that is, the 5th frame is the target entity La 11 Therefore, it can be determined that the target entity La 11 The time region in the continuous image frames is the 1st frame to the 5th frame.
[0155] For another example, for a certain video of continuous image frames, a target entity appears in the 10th frame, and the target entity is recorded as La 31 The target entity La does not exist in the 9th frame. 31 , we can determine that the 10th frame is the target entity La 31Assume that the target entity La 31 After the 10th frame, it appears continuously until the 16th frame. The target entity La does not appear in the 17th frame. 31 , that is, the 16th frame is the target entity La 31 Therefore, it can be determined that the target entity La 31 The time region in the continuous image frames is the 10th frame to the 16th frame.
[0156] In an embodiment of the present invention, determining the starting frame of the target entity in the continuous image frames may include: if a target entity is included in the first frame of the continuous image frames, determining the starting frame of the target entity to be the first frame of the continuous image frames; or, if a target entity is detected in a subsequent frame that has not been successfully matched with any target entity included in the previous frame, determining the unsuccessfully matched target entity in the subsequent frame to be a new target entity, and determining the subsequent frame to be the starting frame of the new target entity.
[0157] In an embodiment of the present invention, determining the end frame of the target entity in the continuous image frames may include: after determining the start frame of a target entity, if it is detected that the target entity satisfies a termination tracking condition, determining the end frame of the target entity to be the previous frame when the target entity satisfies the termination tracking condition.
[0158] In an example, suppose that a target entity appears in the 10th frame of a certain video, denoted as Ln. The target entity has not been successfully matched with all target entities in the previous frame, indicating that the target entity appears for the first time, that is, the target entity is a new target entity, and the 10th frame can be determined as the starting frame of the new target entity Ln. Assume that in the frame after the 10th frame (that is, the 11th frame), the target entity Ln is successfully matched, and then continuously matched successfully to the 25th frame, and the target entity Ln fails to match in the 26th frame, indicating that the target entity Ln meets the termination tracking condition, and the tracking of the target entity Ln can be stopped. At this time, it can be determined that the previous frame (the 25th frame) when the target entity meets the termination tracking condition is the end frame of the target entity Ln. It can be obtained that the time area corresponding to the target entity Ln in this video is from the 10th frame to the 25th frame.
[0159] In summary, the embodiment of the present invention performs target detection on each image frame in the continuous image frames, determines the target entity of the preset entity category in each image frame and the regional coordinates of each target entity; and in each image frame, determines the surrounding entities within the preset range where the regional coordinates of each target entity are located. The entity category and entity position of each surrounding entity near the target entity are used to encode the dimension of the target entity in each preset entity category, and obtain the environmental element coding of the target entity corresponding to each preset entity category. The environmental element coding can reflect the environmental characteristics near the target entity, such as which entities of the preset entity category exist near the target entity. In a frame of image, there may be two or more entities with similar features, but the surrounding environments of these two or more entities with similar features may be different. The embodiment of the present invention introduces environmental element coding to the target entity, and reflects the surrounding environmental characteristics of the target entity in the target coding, which can increase the distinctiveness and recognition between the target entities.
[0160] Based on the environmental element coding of the target entity, the target coding of the target entity is determined, and according to the similarity between the target coding of each target entity in the previous frame and the target coding of each target entity in the next frame, the tracking result of the target entity in the previous frame in the next frame is determined. Since the target coding incorporates the information of the environmental element coding, the surrounding environmental features of the target entity can be reflected. The target coding is used to match the target entities of two adjacent frames. Compared with simple visual features or motion features, the distinctiveness and recognition of the target entity are increased, and the matching errors caused by the similarity of the features of multiple target entities can be avoided, thereby improving the accuracy of multi-target tracking.
[0161] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0162] Device Embodiment
[0163] Reference Figure 4 , shows a structural block diagram of an embodiment of a target tracking device of the present invention, the device may include:
[0164] The target detection module 401 is used to perform target detection on each image frame in the continuous image frames, and determine the target entity of the preset entity category in each image frame and the region coordinates of each target entity;
[0165] A surrounding determination module 402, used to determine surrounding entities within a preset range where the area coordinates of each target entity are located;
[0166] The environment coding module 403 is used to encode the dimensions of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity, so as to obtain the environment element code of each target entity corresponding to each preset entity category;
[0167] A target coding module 404, for determining a target coding of each target entity according to the environmental element coding of each preset entity category corresponding to each target entity;
[0168] The result determination module 405 is used to determine the tracking result of the target entity in the previous frame in the next frame according to the similarity between the target code of each target entity in the previous frame and the target code of each target entity in the next frame.
[0169] Optionally, the surrounding determination module includes:
[0170] A center determination submodule, used to determine the center point of the region coordinates of each target entity in each image frame;
[0171] The entity determination submodule is used to determine that entities located on both sides of the center point whose lateral distance is less than a preset distance are peripheral entities, and the preset distance is determined according to the area width of the current target entity.
[0172] Optionally, the environment encoding module includes:
[0173] A traversal submodule is used to encode each target entity in the dimension of each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category;
[0174] Wherein, the traversal submodule includes:
[0175] A pixel division unit, used to divide a pixel area at a preset distance on both sides of the center point of the target entity into a preset number of pixel blocks to obtain pixel blocks corresponding to the target entity;
[0176] a flag setting unit, for identifying, for each preset entity category, whether each pixel block corresponding to the target entity contains a pixel point of the target entity of the preset entity category, and setting a flag bit of each pixel block according to the identification result; and obtaining an environmental element code of the target entity corresponding to the preset entity category according to the flag bit of each pixel block corresponding to the target entity;
[0177] The traversal unit is used to traverse the preset entity categories to obtain the environmental element codes corresponding to each preset entity category for each target entity.
[0178] Optionally, the flag setting unit includes:
[0179] A first setting subunit, configured to set a flag bit of a first pixel block corresponding to the target entity to a first flag bit, wherein the first pixel block is a pixel block including pixel points of the target entity of the preset entity category;
[0180] The second setting subunit is configured to set a flag bit of a second pixel block corresponding to the target entity to a second flag bit, wherein the second pixel block is a pixel block that does not include pixel points of the target entity of the preset entity category.
[0181] Optionally, the target encoding module includes:
[0182] The first encoding submodule is used to concatenate the environmental element codes corresponding to each preset entity category of each target entity to obtain a target code; or
[0183] A first concatenation submodule is used to concatenate the environment element codes corresponding to each preset entity category of each target entity to obtain an environment code vector of each target entity;
[0184] The second encoding submodule is used to perform vector encoding according to the visual features of each target entity to obtain a visual encoding vector of each target entity;
[0185] The second splicing submodule is used to splice the environment coding vector and the visual coding vector of each target entity to obtain the target coding of each target entity.
[0186] Optionally, the device further comprises:
[0187] The tracking termination module is used to stop tracking the target entity in response to detecting that at least one target entity meets the tracking termination condition.
[0188] Optionally, the device further comprises:
[0189] The time region determination module is used to determine the time region of each target entity in the continuous image frames according to the tracking result of the target entity in the previous frame in the next frame.
[0190] The embodiment of the present invention performs target detection on each image frame in the continuous image frames, determines the target entity of the preset entity category in each image frame and the regional coordinates of each target entity; and in each image frame, determines the surrounding entities within the preset range where the regional coordinates of each target entity are located. The entity category and entity position of each surrounding entity near the target entity are used to encode the dimension of the target entity in each preset entity category, and obtain the environmental element coding of the target entity corresponding to each preset entity category. The environmental element coding can reflect the environmental characteristics near the target entity, such as which entities of the preset entity category exist near the target entity. In a frame of image, there may be two or more entities with similar features, but the surrounding environments of these two or more entities with similar features may be different. The embodiment of the present invention introduces environmental element coding to the target entity, and reflects the surrounding environmental characteristics of the target entity in the target coding, which can increase the distinctiveness and recognition between the target entities.
[0191] Based on the environmental element coding of the target entity, the target coding of the target entity is determined, and according to the similarity between the target coding of each target entity in the previous frame and the target coding of each target entity in the next frame, the tracking result of the target entity in the previous frame in the next frame is determined. Since the target coding incorporates the information of the environmental element coding, the surrounding environmental features of the target entity can be reflected. The target coding is used to match the target entities of two adjacent frames. Compared with simple visual features or motion features, the distinctiveness and recognition of the target entity are increased, and the matching errors caused by the similarity of the features of multiple target entities can be avoided, thereby improving the accuracy of multi-target tracking.
[0192] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0193] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0194] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0195] An embodiment of the present invention provides a device for target tracking, including a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: performing target detection on each image frame in a continuous image frame, determining a target entity of a preset entity category in each image frame and the regional coordinates of each target entity; determining surrounding entities within a preset range where the regional coordinates of each target entity are located; encoding the dimensions of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity, and obtaining an environmental element code corresponding to each preset entity category for each target entity; determining a target code for each target entity according to the environmental element code corresponding to each preset entity category for each target entity; and determining a tracking result of the target entity in the previous frame in the next frame according to the similarity between the target code of each target entity in the previous frame and the target code of each target entity in the next frame.
[0196] Optionally, determining surrounding entities within a preset range of the region coordinates of each target entity includes:
[0197] In each image frame, determine the center point of the region coordinates of each target entity;
[0198] Entities located on both sides of the center point and having a lateral distance less than a preset distance are determined as peripheral entities, and the preset distance is determined according to the area width of the current target entity.
[0199] Optionally, encoding the dimensions of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity to obtain the environmental element coding of each target entity corresponding to each preset entity category includes:
[0200] For each target entity, the target entity is encoded in the dimension of each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category;
[0201] The encoding of the target entity in the dimension of each preset entity category includes:
[0202] Dividing a pixel area at a preset distance on both sides of the center point of the target entity into a preset number of pixel blocks to obtain pixel blocks corresponding to the target entity;
[0203] For each preset entity category, identify whether each pixel block corresponding to the target entity contains a pixel point of the target entity of the preset entity category, and set a flag bit of each pixel block according to the identification result; obtain an environmental element code of the target entity corresponding to the preset entity category according to the flag bit of each pixel block corresponding to the target entity;
[0204] The preset entity categories are traversed to obtain the environmental element codes of each target entity corresponding to each preset entity category.
[0205] Optionally, setting a flag bit of each pixel block according to the recognition result includes:
[0206] Setting a flag bit of a first pixel block corresponding to the target entity to a first flag bit, wherein the first pixel block is a pixel block including pixel points of the target entity of the preset entity category;
[0207] The flag bit of the second pixel block corresponding to the target entity is set as the second flag bit, the second pixel block being a pixel block that does not include pixel points of the target entity of the preset entity category.
[0208] Optionally, determining the target code of each target entity according to the environmental element code of each preset entity category corresponding to each target entity includes:
[0209] The environmental element codes of each target entity corresponding to each preset entity category are concatenated to obtain the target code; or,
[0210] The environment element codes of each target entity corresponding to each preset entity category are concatenated to obtain the environment code vector of each target entity;
[0211] Perform vector encoding according to the visual features of each target entity to obtain a visual encoding vector of each target entity;
[0212] The environment encoding vector and visual encoding vector of each target entity are concatenated to obtain the target encoding of each target entity.
[0213] Optionally, the device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:
[0214] In response to detecting that at least one target entity meets a tracking termination condition, tracking of the target entity is stopped.
[0215] Optionally, the device is further configured to execute, by one or more processors, the one or more programs including instructions for performing the following operations:
[0216] According to the tracking result of the target entity in the previous frame in the subsequent frame, the time region of each target entity in the continuous image frames is determined.
[0217] Figure 58 is a block diagram of a device 800 for target tracking according to an exemplary embodiment. For example, the device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0218] Reference Figure 5 , the device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0219] The processing component 802 generally controls the overall operation of the device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0220] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0221] The power supply component 806 provides power to the various components of the device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.
[0222] The multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0223] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the device 800 is in an operating mode, such as a call mode, a recording mode, and a voice information processing mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0224] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0225] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the device 800, and the sensor assembly 814 can also test the position change of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, the orientation or acceleration / deceleration of the device 800, and the temperature change of the device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0226] The communication component 816 is configured to facilitate wired or wireless communication between the device 800 and other devices. The device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency information processing (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0227] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0228] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0229] Figure 6 1 is a schematic diagram of the structure of the server in some embodiments of the present invention. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1922 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1922 can be configured to communicate with the storage medium 1930 and execute a series of instruction operations in the storage medium 1930 on the server 1900.
[0230] The server 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958, one or more keyboards 1956, and / or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
[0231] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to perform Figure 1 The target tracking method shown.
[0232] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by a processor of a device (server or terminal), enables the device to execute a target tracking method, the method comprising: performing target detection on each image frame in a series of image frames, determining a target entity of a preset entity category in each image frame and the regional coordinates of each target entity; determining surrounding entities within a preset range where the regional coordinates of each target entity are located; encoding the dimensions of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity, and obtaining an environmental element code corresponding to each preset entity category for each target entity; determining a target code for each target entity according to the environmental element code corresponding to each preset entity category for each target entity; and determining a tracking result of the target entity in the previous frame in the next frame according to the similarity between the target code of each target entity in the previous frame and the target code of each target entity in the next frame.
[0233] Those skilled in the art will readily appreciate other embodiments of the invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0234] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
[0235] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
[0236] The target tracking method, the target tracking device and the device for target tracking provided by the present invention are introduced in detail above. The principle and implementation mode of the present invention are explained by using specific examples in this article. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation mode and the scope of application. In summary, the content of this specification should not be understood as limiting the present invention.
Claims
1. A target tracking method, characterized in that: The method comprises: Performing target detection on each image frame in the continuous image frames to determine a target entity of a preset entity category in each image frame and a region coordinate of each target entity; Determine surrounding entities within a preset range where the regional coordinates of each target entity are located; According to the entity category and entity position of each surrounding entity, encode the dimension of each target entity in each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category; Determine the target code of each target entity according to the environmental element code corresponding to each preset entity category of each target entity; Determining a tracking result of the target entity in the previous frame in the next frame according to a similarity between a target code of each target entity in the previous frame and a target code of each target entity in the next frame; The encoding of each target entity in the dimension of each preset entity category according to the entity category and entity position of each surrounding entity to obtain the environmental element coding of each target entity corresponding to each preset entity category includes: For each target entity, the target entity is encoded in the dimension of each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category; The encoding of the target entity in the dimension of each preset entity category includes: Dividing a pixel area at a preset distance on both sides of the center point of the target entity into a preset number of pixel blocks to obtain pixel blocks corresponding to the target entity; For each preset entity category, identify whether each pixel block corresponding to the target entity contains a pixel point of the target entity of the preset entity category, and set a flag bit of each pixel block according to the identification result; obtain an environmental element code of the target entity corresponding to the preset entity category according to the flag bit of each pixel block corresponding to the target entity; The preset entity categories are traversed to obtain the environmental element codes of each target entity corresponding to each preset entity category.
2. The method according to claim 1, characterized in that: The step of determining surrounding entities within a preset range where the regional coordinates of each target entity are located includes: In each image frame, determine the center point of the region coordinates of each target entity; Entities located on both sides of the center point and having a lateral distance less than a preset distance are determined as peripheral entities, and the preset distance is determined according to the area width of the current target entity.
3. The method according to claim 1, characterized in that The step of setting the flag bit of each pixel block according to the recognition result includes: Setting a flag bit of a first pixel block corresponding to the target entity to a first flag bit, wherein the first pixel block is a pixel block including pixel points of the target entity of the preset entity category; The flag bit of the second pixel block corresponding to the target entity is set as the second flag bit, the second pixel block being a pixel block that does not include pixel points of the target entity of the preset entity category.
4. The method according to claim 1, characterized in that: Determining the target code of each target entity according to the environmental element code of each preset entity category corresponding to each target entity includes: The environmental element codes of each target entity corresponding to each preset entity category are concatenated to obtain the target code; or, The environment element codes of each target entity corresponding to each preset entity category are concatenated to obtain the environment code vector of each target entity; Perform vector encoding according to the visual features of each target entity to obtain a visual encoding vector of each target entity; The environment encoding vector and visual encoding vector of each target entity are concatenated to obtain the target encoding of each target entity.
5. The method according to claim 1, characterized in that The method further comprises: In response to detecting that at least one target entity meets a tracking termination condition, tracking of the target entity is stopped.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: According to the tracking result of the target entity in the previous frame in the subsequent frame, the time region of each target entity in the continuous image frames is determined.
7. A target tracking device, characterized in that: The device comprises: A target detection module is used to perform target detection on each image frame in the continuous image frames, and determine the target entity of a preset entity category in each image frame and the regional coordinates of each target entity; A surrounding determination module, used to determine surrounding entities within a preset range where the regional coordinates of each target entity are located; An environmental coding module is used to encode the dimensions of each target entity in each preset entity category according to the entity category and entity position of each surrounding entity, so as to obtain the environmental element coding of each target entity corresponding to each preset entity category; A target coding module, used to determine the target coding of each target entity according to the environmental element coding of each target entity corresponding to each preset entity category; A result determination module, used to determine the tracking result of the target entity in the previous frame in the next frame according to the similarity between the target code of each target entity in the previous frame and the target code of each target entity in the next frame; Wherein, the environment coding module includes: A traversal submodule is used to encode each target entity in the dimension of each preset entity category to obtain the environmental element code of each target entity corresponding to each preset entity category; Wherein, the traversal submodule includes: A pixel division unit, used to divide a pixel area at a preset distance on both sides of the center point of the target entity into a preset number of pixel blocks to obtain pixel blocks corresponding to the target entity; a flag setting unit, for identifying, for each preset entity category, whether each pixel block corresponding to the target entity contains a pixel point of the target entity of the preset entity category, and setting a flag bit of each pixel block according to the identification result; and obtaining an environmental element code of the target entity corresponding to the preset entity category according to the flag bit of each pixel block corresponding to the target entity; The traversal unit is used to traverse the preset entity categories to obtain the environmental element codes corresponding to each preset entity category for each target entity.
8. The device according to claim 7, characterized in that The surrounding determination module includes: A center determination submodule, used to determine the center point of the region coordinates of each target entity in each image frame; The entity determination submodule is used to determine that entities located on both sides of the center point whose lateral distance is less than a preset distance are peripheral entities, and the preset distance is determined according to the area width of the current target entity.
9. The device according to claim 7, characterized in that The target encoding module comprises: The first encoding submodule is used to concatenate the environmental element codes corresponding to each preset entity category of each target entity to obtain a target code; or A first concatenation submodule is used to concatenate the environment element codes corresponding to each preset entity category of each target entity to obtain an environment code vector of each target entity; The second encoding submodule is used to perform vector encoding according to the visual features of each target entity to obtain a visual encoding vector of each target entity; The second splicing submodule is used to splice the environment coding vector and the visual coding vector of each target entity to obtain the target coding of each target entity.
10. The device according to claim 7, characterized in that The device also includes: The tracking termination module is used to stop tracking the target entity in response to detecting that at least one target entity meets the tracking termination condition.
11. The device according to any one of claims 7 to 10, characterized in that: The device also includes: The time region determination module is used to determine the time region of each target entity in the continuous image frames according to the tracking result of the target entity in the previous frame in the next frame.
12. A device for target tracking, characterized in that: The invention comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs include instructions for performing the target tracking method as claimed in any one of claims 1 to 6.
13. A machine-readable medium having instructions stored thereon, which, when executed by one or more processors of a device, causes the device to execute the target tracking method according to any one of claims 1 to 6.
