Method for associating objects in a video

Abstract

The invention relates to a computer-implemented method (300) for associating objects in a video (V), the video (V) comprising consecutive frames (F n‑1 , F n ), the method comprises: obtaining a previous frame (F n‑1 ) to determine the first target proposal area information of the current frame (F n ), wherein the first and second target proposal region information at least indicate the appearance metric, spatial position and detection probability of each target proposal region of the corresponding frame, by at least combining the previous frame (F n‑1 ) is associated to the current frame (F n ) to associate objects in the video (V) using a distance metric (D) calculated based on appearance metrics, spatial location, and detection probability. n,n‑1 ) to associate the target proposal region.

Description

Technical Field

[0001] The present invention relates to a method for associating objects in a video and further relates to a computer for executing the method. Background Art

[0002] A visual object detector (e.g., in the form of a computer or video processor) can be used to automatically identify and localize objects that may appear in a video frame. An object detector typically provides the location of the object, the size of the object, and the type of the object, or any combination of these items. The type of object can be identified as an object class, such as a vehicle, a person, an animal, or a building. Automatically identifying and localizing objects in a frame is a necessary step in many augmented reality (AR) and security applications. Some state-of-the-art object detectors are based on convolutional neural networks (CNNs).

[0003] An example can be found in S. Ren, K. He, R. Girshick, and J. Sun, “ Faster R-CNN: Towards real-time object detection with region proposal networks ” (IEEE Trans. Pattern Analysis and Machine Intelligence, 2017). Another example can be found in “ SSD: Single shot multibox detector ” (in Proc. European Conference Computer Vision, 2016). Another example can be found in J. Redmon and A. Farhadi, “ YOLO9000: Better, faster, stronger ” (In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2017).

[0004] A disadvantage of such conventional systems is that they may suffer from object identity or tracklet identity switching. This may typically occur when two or more objects are close together (e.g., two football players fighting for the ball). This is due to the fact that such systems typically apply visual object detectors individually on each of the successive or consecutive frames in the video, thereby ignoring the temporal relationship between objects in consecutive frames; or the fact that the visual scene evolves slowly from one frame to another.

[0005] A naive attempt to improve the accuracy of conventional object detectors using information shared between frames is to smooth the detection probabilities and proposed region coordinates before applying the decision threshold in the current frame. The drawback of the above-mentioned method of smoothing the detection probabilities is that it has low accuracy and may not be accurate enough. For example, if the detection probability of an object drops below a hard detection probability threshold for a few frames, the object may still be easily missed despite the fact that the visual appearance of the object across frames is visually almost unchanged.

[0006] Therefore, there is a need for improved methods and computers.

[0007] Purpose of the Invention

[0008] It is an aim of embodiments of the present invention to provide a solution which alleviates or resolves the above mentioned disadvantages. Summary of the invention

[0009] The above mentioned objects are achieved by the subject matter described herein. Further advantageous implementation forms of the invention are described herein.

[0010] According to a first aspect of the present invention, the above-mentioned purpose is achieved by a computer-implemented method for associating targets in a video, wherein the video includes consecutive frames, and the method includes: obtaining first target proposal area information of a previous frame; determining second target proposal area information of a current frame, wherein the first and second target proposal area information at least indicate the appearance metric, spatial position and detection probability of each target proposal area of ​​the corresponding frame; associating targets in the video by associating at least a first set of target proposal areas of the previous frame to a second set of target proposal areas of the current frame, wherein the target proposal areas are associated using a distance metric calculated based on the appearance metric, spatial position and detection probability.

[0011] At least one advantage of this aspect of the present disclosure is that object detection accuracy is improved by associating objects in consecutive frames using a distance metric based on an appearance metric, spatial location, and a detection probability calculation.

[0012] According to a second aspect of the present invention, the above mentioned object is achieved by a computer configured to execute the method according to the first aspect.

[0013] The scope of the present invention is defined by the claims, which are incorporated into this section by reference. By considering the following detailed description of one or more embodiments, those skilled in the art will gain a more complete understanding of embodiments of the present invention and an appreciation of their additional advantages. Reference will be made to the attached sheet of the figure that will first be briefly described. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1Scenarios for associating objects in a video are described in accordance with one or more embodiments.

[0015] Figure 2 An example of a conventional method for detecting a target is described.

[0016] Figure 3 A method for associating objects in a video is shown in accordance with one or more embodiments.

[0017] Figure 4 Calculation of a distance metric in accordance with one or more embodiments is described.

[0018] Figure 5A Distance metrics according to one or more embodiments are described.

[0019] Figure 5B Mapping of target suggestion regions in accordance with one or more embodiments is described.

[0020] Figure 6 Details of a computer are shown in accordance with one or more embodiments.

[0021] By considering the following detailed description of one or more embodiments, those skilled in the art will gain a more complete understanding of embodiments of the present invention and an appreciation of additional advantages thereof.It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more figures of the accompanying drawings. DETAILED DESCRIPTION

[0022] Generally speaking, unless clearly given and / or different meanings are implied from the context in which the term is used, all terms used herein are interpreted according to their common meanings in the relevant technical field. Unless explicitly stated otherwise, all mentioned one / one / the element, equipment, assembly, parts, steps, etc. are interpreted as at least one instance of the element, equipment, assembly, parts, steps, etc. in an open manner. Unless a step is explicitly described as after or before another step and / or when it is implied that a step must be after or before another step, the steps of any method disclosed herein do not have to be performed in the exact order disclosed. In any appropriate case, any feature of any embodiment in the embodiments disclosed herein may be applied to any other embodiment. Similarly, any advantage of any embodiment in these embodiments may be applicable to any other embodiment, and vice versa. Other purposes, features and advantages of the attached embodiments will emerge from the following description.

[0023] In the present disclosure, the term "target suggestion area" refers to an area indicating a range within which a suggested target exists. The area may have any suitable form, such as a rectangle, an ellipse, a circle, or an arbitrary shaped polygon with associated center coordinates, for example, at or near the center of gravity.

[0024] In one example, target suggestion regions are received from a neural network algorithm, each target suggestion region including a possible target or candidate target. The size of the candidate target may be defined by the target suggestion region, for example in the form of a bounding box. The bounding box / target suggestion region may surround the candidate target. The horizontal and vertical dimensions of the target suggestion region may be the width and height of the bounding box. Alternatively or additionally, the size of the selected target (e.g., each selected target suggestion region including the target) may be defined by a bounding box surrounding the selected target, and the target suggestion region coordinates may be the width and height of the bounding box. The coordinates may alternatively or additionally be the coordinates of the center of the bounding box.

[0025] In the present disclosure, the term “detection probability” refers to the possibility or probability that a detected proposed area includes a target.

[0026] In this disclosure, the term "frame" refers to a frame, a frame in a video, an image, or a video frame. In other words, individual consecutive images constitute a video when shown one after another.

[0027] The proposed method provides a solution to the above-listed drawbacks of conventional solutions. This is achieved by performing high-level data association based on location, size, detection probability, and similarity measures (based on object appearance) to connect objects in frame n-1 to frame n.

[0028] Even if objects are close together and have the same size, differences in appearance help distinguish objects, perform correct data association, and maintain track IDs. Appearance can be captured by computing the color histogram of object proposals in consecutive frames, and is also used to increase the detection probability of proposal regions when they have similar appearance across frames.

[0029] The advantage of the proposed solution is at least an increase in object detector accuracy as well as stability of track ID. The proposed algorithm can be run in real-time as a post-processor of a visual object detector, which allows the technique to be reused with all types of modern detectors.

[0030] In one example, a target proposal region is identified or detected using a neural network, which relies on a region proposal algorithm to assume the location of a target within a frame. For example, under the assumption that all targets of interest share common visual properties that distinguish them from the background, a method is designed or trained that outputs a set of proposed regions that may contain targets within a frame. These methods tend to calculate a large number of candidate targets / target proposal regions for each frame, such as 4,800 candidate targets. The candidate targets can be represented as a matrix. Each row of the matrix can represent a candidate target, and the column specifies the position and size of the candidate target. The position and size can be defined relative to the target proposal region or bounding box that surrounds the candidate target. The column can define the height and width of the bounding box, and the position can be an anchor point for constructing the bounding box, such as the center point or corner of the bounding box. Columns are also included to provide a target detection probability or confidence score for each candidate target identified, that is, the calculated detection probability that the content of the target proposal region is correctly identified as a target. The set of target proposal regions / candidate targets contained in the matrix can then be simply thresholded to provide a list of detected targets. Specifically, the detection probability provided for each candidate target / target proposal region is compared with a threshold value T, and if its detection probability exceeds the threshold value T, the candidate target is detected as a target and / or added to a target list. Then, a list of detected targets may be output. An overlay may be provided when displaying the frame, for example, to indicate all or some of the targets from the output list, such as by showing the outline of a bounding box, or by displaying a marker, target identity, or track identity below, above, or beside the detected target.

[0031] In the present disclosure, the term "appearance metric" refers to a metric of visual appearance. Two different proposal regions comprising visual representation / image pixels having similar or identical appearance metrics are considered similar, i.e., similar appearance metrics indicate visual similarity. An example of an appearance metric is a color histogram calculated using color values ​​of frame pixels included in the proposal region, e.g., a histogram calculated based on RGB values ​​of pixels included in the proposal region.

[0032] In the present disclosure, the term “spatial position” refers to a position of or with respect to a captured frame, for example, rows and columns of pixels included in the frame.

[0033] Figure 1 Scenarios for associating objects in a video are described in accordance with one or more embodiments. Figure 1 Description includes the previous frame F n-1 and the current and consecutive frames F n It is to be understood that the video V may include previous frames.

[0034] For example, a video or video clip is received by a computer or a video processor. Video can be received in different ways. For example, a video can be provided as a video feed, such as a streaming video feed. The video feed can be obtained from a live recording and can be optionally obtained in real time. Alternatively or additionally, a video clip can be received by retrieving a previously recorded and stored video clip. For example, the video clip can be recorded on a storage medium such as a DVD or a Blu-ray disc, or recorded on a computer memory device such as a hard drive, a flash drive, a memory stick, or a memory card. The video can be a complete video, such as a movie or a show episode. Alternatively, the video can be an excerpt from a longer video. The video clip can be received as part of a longer video, and a computer can be used to segment the longer video and create a shorter video. For example, a continuous shot from a sports event can be received by a computer, and then the computer creates a shorter video corresponding to the highlights of the sports event. The video clip can even be just a segment of a game in a sports event. By way of example only, in this article, a video is described as a segment of a game in a soccer match, such as a segment of a game immediately before and including a goal score in a soccer match.

[0035] The previous frame F is obtained, for example, by retrieving information from memory n-1 The first target proposal area information. The first target proposal area information may generally include a first set of target proposal areas 111, 121, 131, 141. The first target proposal area information may further include a detection probability of each of the target proposal areas 111, 121, 131, 141. The first target proposal area information may further include an appearance metric of each of the target proposal areas 111, 121, 131, 141. The first target proposal area information may further include a spatial position of each target proposal area, such as a center coordinate of the target proposal area. Optionally, only a subset of N, for example, 25 target proposal areas with the highest detection probability is obtained. In other words, target-related target proposal areas with the highest detection probability are selected, for example, 25 target proposal areas with the highest detection probability.

[0036] As can be seen from the figure, only three regions 111, 121, 131 of the object proposal regions include actual objects 110, 120, and 130. The object proposal region 141 can be regarded as a "false detection".

[0037] Then, determine the current frame F nThe second target proposal region information may generally include a second set of target proposal regions 151, 161, 171, 181 and a corresponding spatial position and detection probability of each of the target proposal regions 151, 161, 171, 181. The spatial position may, for example, include the center coordinates of the target proposal region. A neural network is generally used to identify or detect the target proposal regions 151, 161, 171, 181, which relies on a region proposal algorithm to assume that the current frame F n The target location within the current frame F is determined by the following equation: n The appearance metric of each of the target proposal regions 151, 161, 171, 181 is calculated, and the appearance metric is added to the second target proposal region information. An example of the appearance metric is a color histogram calculated using the color values ​​of the frame pixels included in the proposal region, such as a histogram calculated based on the RGB values ​​of the pixels included in the proposal region. Optionally, only a subset of N, for example 25, target proposal regions with the highest detection probability is determined.

[0038] As can be seen from the figure, only three regions 151, 161, 171 of the object proposal regions include actual objects 150, 160, and 170. The object proposal region 181 can be regarded as a "false detection".

[0039] By at least the previous frame F n-1 The first set of target proposal regions 111, 121, 131, 141 is associated with the current frame F n The second set of object proposal regions 151, 161, 171, 181 is used to associate objects in the video V. The object proposal regions may be associated using a distance metric calculated based on appearance metrics, spatial locations, and detection probabilities.

[0040] It can be further based on the distance metric D n,n-1 The minimum distance will be the previous frame F n-1 The first set of target proposal regions is mapped to the current frame F n A second set of target proposal regions is used to associate targets in the video.

[0041] The current frame F can be further adjusted n This typically involves increasing or shifting the detection probability of each associated object proposal region so that, for example, nThe detection probability of the target in the previous frame F n-1 In scenarios where the probability of detection is temporarily reduced, the risk of losing or missing the target between frames is reduced.

[0042] Associating an object in a video may further include determining a current frame F having a detection probability above a threshold T. n The associated and / or mapped target proposal regions are used to detect the current frame F n goal.

[0043] In one example, the previous frame F n-1 The target 110 in the current frame F is associated with n The target in is 150, because the combined distance metric D n,n-1 It indicates that for the object proposal region 111 associated with the object 110 and the object proposal region 151 associated with the object 150, the appearance metric, spatial position and detection probability are most similar. Then, it can be determined that the object 110 and the object 150 represent the previous frame F n-1 and the current frame F n In addition, this may involve assigning the same track identity to target 110 and target 150, for example, to the previous frame F. n-1 The target 110 and the current frame F n 150 in the example are assigned the same name of the football player. It can be further determined that the target suggestion region 121 associated with the target 120 is associated with the target suggestion region 161 associated with the target 160. It can be further determined that the target suggestion region 131 associated with the target 130 is associated with the target suggestion region 171 associated with the target 170. It can be further determined that the target suggestion region 141 is not associated with the target suggestion region 181. In other words, the combined distance metric D n,n-1 The designated object suggestion region 141 and the object suggestion region 181 include or represent different objects.

[0044] Figure 2 An example of a conventional method for detecting a target is described. In this example, a frame F is received. n-1 、F n Then, each frame is detected individually, i.e., operated on a frame-by-frame basis, and each frame is analyzed to detect the target O n . Determine or detect multiple object proposal regions along with their corresponding spatial locations and object detection probabilities.

[0045] Detecting objects O is done by simply comparing each detection probability to a “hard” threshold T n In other words, all object proposals with an object detection probability higher than a threshold T are detected as objects.

[0046] This method has the disadvantage that the target may be lost between frames, for example, in frame F n-1 The target is detected in frame F, but n Another disadvantage is that when targets are close and / or cross paths, the target or track identities may be swapped.

[0047] Figure 3 A method 300 for associating a proposed region representing an object in a video is shown, which is performed by a computer 600 according to one or more embodiments. A method for associating a proposed region including consecutive frames F is provided. n-1 、F n A computer-implemented method 300 for a target in a video V. The method comprises:

[0048] Step 310: Get the previous frame F n-1 The first target proposed region information at least indicates the previous frame F n-1 The appearance metric, spatial position and detection probability of each object proposal region of the first object proposal region can be obtained by, for example, reading from the memory 615 of the computer.

[0049] In an embodiment, the method includes an initialization phase when receiving a first frame, comprising performing steps 320 and 336 on the first frame of the video to obtain a first previous frame F n-1 The first target suggestion area information.

[0050] Step 320: Determine the current frame F n The second target proposed region information at least indicates the current frame F n The corresponding appearance metric, spatial location, and detection probability of each object proposal region.

[0051] Combination Figure 1 Determining second target suggestion area information is further described.

[0052] In one example, the corresponding appearance metric is determined as a histogram calculated based on the RGB values ​​of the pixels included in the proposal region. The spatial position is determined as the center coordinates of the target proposal region. The detection probability is obtained from a neural network algorithm.

[0053] In the present disclosure, color histograms are used to compute / determine the appearance metric and thus model the visual object appearance and compute the similarity between objects across frames. Alternatively, a feature descriptor such as Histogram of Oriented Gradients (HOG) can be used to compute / determine the appearance metric.

[0054] The RGB color space may be used to calculate / determine the appearance metric of the object. Alternatively, color spaces such as Lab and HSV may be used to reduce the impact of image intensity that affects the histogram.

[0055] As combined Figure 4 As further described, the appearance distance d(c n ,c n-1 ) can be calculated using the normalized root mean square error, but alternatively it can be calculated / determined using histogram intersection, Bhattacharyya distance, or chi-square.

[0056] Optional step 325: Select the object with the highest detection probability. In some cases, thousands of object proposals are determined / detected, and this can be done by considering only the objects from the previous frame F that have the highest detection probability. n-1 The number of n-1 object proposal regions and the n The number of n The object proposal region is used to reduce the computational complexity.

[0057] The maximum number of objects in a frame depends on the application. In the case of a detector for sports videos (e.g., football), it can be set to L M =25.

[0058] In one example, the current frame F is provided n A large number of candidate targets / target proposal regions, such as 4800 candidate targets, are selected. The 4800 candidate targets are then sorted by target detection probability, with the highest target detection probability first. Then, 25 candidate targets / target proposal regions with the highest target detection probability are selected for further processing in the method, and the remaining candidate targets / target proposal regions are ignored.

[0059] Step 330: By at least the previous frame F n-1 The first set of target proposal regions 111, 121, 131, 141 is associated with the current frame F n The second set of target proposal regions 151, 161, 171, 181 is used to associate the target in the video V. In one embodiment, a distance metric D calculated based on appearance metric, spatial position and detection probability is used. n,n-1 to associate the target proposal area.

[0060] In one example, the targets with the smallest distance metric are considered identical. Figure 1 A more detailed example of associating the objects in video V is provided.

[0061] In an embodiment, associating objects in the video V is performed by mapping object proposal regions, adjusting the detection probability of each mapped object proposal region, and then detecting objects with a detection probability above a threshold.

[0062] This has the advantage of improving object detection accuracy by reducing the likelihood of objects being lost or missed between consecutive frames.

[0063] In one embodiment, the step of associating the target further comprises the additional steps of:

[0064] Optional step 332: Based on the distance metric D n,n-1 The minimum distance of the previous frame F n-1 The first set of target proposal regions is mapped to the current frame F n The second set of target proposal regions.

[0065] This can be combined with Figure 5B Executed as further described.

[0066] Optional step 334: Adjust the current frame F n The detection probability of each mapped object proposal region.

[0067] In one embodiment, associating the target in the video is further performed by: if the corresponding appearance distance metric is below a second threshold , then adjust the current frame F n The detection probability of each mapped object proposal region.

[0068] In one embodiment, adjusting the probability of detection further comprises determining a coefficient as:

[0069]

[0070] in and are two mapped object proposals (one from frame F n And one from frame F n-1 )’s appearance metric. max and α min It's the forgetting factor. and is the low probability threshold that makes the proposed region eligible for detection probability adjustment.

[0071] In one embodiment, adjusting the detection probability further includes increasing the detection probability of a given target proposal region, for example to prevent the loss of existing targets due to fluctuations in the detection probability in the current frame. , the detection probability is increased by determining the updated detection probability as follows:

[0072]

[0073] in and are the detection probabilities of the two mapped object proposals (one from frame n and one from frame n-1).

[0074] In an embodiment, the second threshold Can be set to =0.1.

[0075] In an embodiment, wherein and .

[0076] In an embodiment, the forgetting factor can be set to and .

[0077] Optional step 336: By determining the current frame (F n ) to detect the current frame (F n ) goals.

[0078] In an embodiment, the detection threshold can be set to .

[0079] In some embodiments, the target is tracked in the sense that the identity follows the target at least as long as the target is within a frame of the video v.

[0080] In another embodiment, the method further comprises the following steps:

[0081] Optional step 340: Tracking identity from the previous frame F n-1 The targets 110, 120, 130 detected in the current frame F are propagated to n Targets 150, 160, 170 are detected in FIG.

[0082] In one embodiment, the previous frame F n-1 The target proposal area and the current frame F n The distance measure D between target proposal regions n,n-1 Calculated as: D n,n-1 =d(X n ,X n-1 )+d(c n ,c n-1 ), where d(X n ,X n-1 ) is a combined spatial probability distance metric, and d(c n ,c n-1 ) is the appearance distance metric.

[0083] In one embodiment, the distance metric (D n,n-1 ) is calculated as a weighted sum of the combined spatial probability distance metric and the appearance distance metric. Figure 4 Computing the distance metric is further described.

[0084] In order to provide the reader with a better understanding of possible applications of the present disclosure, a use case embodiment of the present disclosure is presented below:

[0085] Initialization (first frame in the video sequence) :

[0086] 1) In the initial frame, select L with the highest probability M Suggested areas.

[0087] 2) Passing the detection threshold Accept region proposals as detected objects.

[0088] 3) Calculate the RGB color histogram c0 of those targets.

[0089] 4) Assign tracking IDs to all detected targets.

[0090] Repeat for all subsequent frames :

[0091] 5) In the current frame (with index n), pick the L with the highest probability M Suggested areas.

[0092] 6) Calculate the RGB color histogram c for the selected proposal in the current frame n .

[0093] 7) Calculate the current frame n and the previous frame F n-1 The distance measure between all selected region proposals in L M L M Distance D n,n-1 ), where the distance metric is calculated as: D n,n-1 =d(X n ,X n-1 )+d(c n ,c n-1 ),

[0094] Where d(c n ,c n-1 ) is the distance between the color histograms of the selected region proposals in the past and current frames (it is a L M L M ),

[0095] Similarly, d(Xn ,X n-1 ) is the matrix capturing the distance between the detection probability and spatial coordinates of the selected proposal region.

[0096] 8) Use the greedy algorithm (eliminating the proposed region pairs with the smallest distance one by one) to find the frame F n and frame F n-1 Data association is accomplished by minimizing the distance between proposals in the mapping.

[0097] 9) If the mapped suggestion has a value below a certain threshold If the histogram distance is , the coefficients used to update the detection probability before thresholding (used in the next algorithm step) are modified as follows:

[0098]

[0099] in and are two proposed regions connected by a data association algorithm (one from frame F n And one from frame F n-1 )’s color histogram.

[0100] 10) Increase the detection probability of a given proposal region to prevent losing existing targets due to fluctuations in detection probability in the current frame,

[0101] if

[0102]

[0103] in and are two proposed regions connected by a data association algorithm (one from frame F n And one from frame F n-1 )’s detection probability.

[0104] 11) The next step is to accept values ​​above the hard threshold As detected objects, the proposals are then mapped, the IDs from the previous frame are propagated. For unmatched proposals, new IDs are assigned.

[0105] 12) Convert the n The ID, calculated region proposal, and histogram are stored in a buffer for the next frame F n+1 middle.

[0106] The term X used above can consist of the bounding box coordinates of the proposed region, e.g., the center (x, y), width (w) and height (h) of the bounding box, and the detection probability p of this region. In this case, for each proposed region, a 5-dimensional vector can be formed:

[0107] X={x,y,w,h,p}

[0108] The total distance between proposals from consecutive frames (algorithm step 7) can have different weights:

[0109]

[0110] in Preferably greater than .

[0111] Figure 4 The distance metric D according to one or more embodiments is described as follows: n,n-1 Calculation. Figure 4 Description includes the previous frame F n-1 and the current and consecutive frames F n Video V. As combined Figure 1 The previous frame F n-1 The first target proposal region information is obtained and generally includes a first set of target proposal regions 111, 121, 131, 141. The first target proposal region information may further include corresponding detection probabilities, appearance metrics, and spatial locations of the first set of target proposal regions 111, 121, 131, 141. Then, determine the current frame F n The second target proposal region information generally includes a second set of target proposal regions 151, 161, 171, 181 and a corresponding spatial position, detection probability, and appearance metric of each of the target proposal regions 151, 161, 171, 181.

[0112] Then, the distance metric D between each proposal region in the first set of target proposal regions 111, 121, 131, 141 and each proposal region in the second set of target proposal regions 151, 161, 171, 181 is calculated. n,n-1 This is achieved through Figure 4 For the sake of clarity, only the distance d between a proposed region 111 and the set of target proposed regions 151, 161, 171, 181 is shown. 111_151 ,d 111_161 ,d 111_171 and d 111_181 .

[0113] In one example, the distance metric may be calculated as the previous frame F n-1and the current frame F n The detection probability, appearance metric, and the sum of the Euclidean distance between spatial locations or any one of the Euclidean metrics.

[0114] In one example, the distance metric d 111_151 is calculated as

[0115] d 111_151 = spatial distance + detection probability distance + appearance distance

[0116]

[0117] where X represents the coordinates of the object proposal region, p represents the object detection probability of the object proposal region, and c represents the appearance metric of the object proposal region.

[0118] In another example, the distance metric d 111_151 is weighted and calculated as

[0119]

[0120] In one embodiment, the object proposal region is formed as a rectangle with two-dimensional center coordinates (x, y), width w, and height h, where the spatial probabilistic distance metric is calculated based on a vector formed by X=[x, y, w, h, p], where p is the detection probability of the object proposal region.

[0121] Figure 5A The distance metric D according to one or more embodiments is described as follows: n,n-1 . refer to Figure 4 , the distance metrics may be stored or arranged in a matrix data structure. As can be seen in the figure, the distance metrics for the target proposal region 111 are arranged in the first column, the target proposal region 121 is arranged in the second column, and so on.

[0122] Figure 5B The mapping of target proposal regions according to one or more embodiments is illustrated. 121_161 Including the fact that the minimum distance of the distance metric in the second column, the previous frame (F n-1 ) is mapped to the current frame (F n ) suggestion region 161. In other words, it is determined that the target 120 included in the target suggestion region 121 represents the same target as the target 160 included in the target suggestion region 161.

[0123] Figure 6Details of a computer 600 according to one or more embodiments are shown. The computer 600 may take the form of a selection of any one of an image processing device, one or more servers, one or more clouds or virtual servers. The computer 600 may include a processing circuit 612, which may be optionally communicatively coupled to a communication interface 604 for wired and / or wireless communication. In addition, the computer 600 may further include at least one optional antenna (not shown in the figure). The antenna may be coupled to a transceiver of the communication interface 604 and configured to transmit and / or transmit and / or receive wireless signals in a wireless communication system. In one example, the processing circuit 612 may be any one of a selection of multiple processors and / or processor modules and / or central processing units and / or processors configured to cooperate with each other. In addition, the computer 600 may further include a memory 615. The memory 615 may contain instructions executable by the processing circuit to perform any one of the methods and / or method steps described herein.

[0124] The communication interface 604 (e.g., a wireless transceiver and / or a wired / wireless communication network adapter) is configured to send and / or receive data values ​​or parameters as signals to and / or from the processing circuit 612, to and / or from other external nodes (e.g., a video streaming server (not shown)). In an embodiment, the communication interface communicates directly between nodes or via a communication network.

[0125] In one or more embodiments, the computer 600 may further include an input device 617 configured to receive input or instructions from a user and send a user input signal indicative of the user input or instructions to the processing circuit 612 .

[0126] In one or more embodiments, the computer 600 may further include a display 618 configured to receive a display signal indicating a presentation target, such as a textual or graphical user input target, from the processing circuit 612 and display the received signal as a target, such as a textual or graphical user input target.

[0127] In one embodiment, the display 618 is integrated with the user input device 617 and is configured to receive a display signal indicating a presentation target such as a textual or graphical user input target from the processing circuit 612 and display the received signal as a target such as a textual or graphical user input target, and / or is configured to receive input or indications from a user and send a user input signal indicating the user input or indication to the processing circuit 612.

[0128] In an embodiment, processing circuit 612 may be communicatively coupled to memory 615 and / or communication interface 604 and / or input device 617 and / or display 618 and / or one or more sensors (not shown).

[0129] In an embodiment, the communication interface and / or transceiver 604 communicates using wired and / or wireless communication techniques. In an embodiment, the one or more memories 615 may include a hard RAM disk drive, a floppy disk drive, a tape drive, an optical drive, a CD or DVD drive (R or RW) or other removable or fixed media drive options.

[0130] In further embodiments, the computer 600 may further include and / or be coupled to one or more additional sensors (not shown) configured to receive and / or obtain and / or measure physical properties related to the computer or the computer's environment and to send one or more sensor signals indicating the physical properties to the processing circuit 612.

[0131] It is understood that the computer includes any suitable combination of hardware and / or software required to perform the tasks, features, functions, and methods disclosed herein. In addition, although the components of the computer are depicted as a single box located within a larger box or nested within multiple boxes, in practice, the computer may include multiple different physical components that make up the single illustrated component (e.g., the memory 615 may include multiple separate hard drives and multiple RAM modules).

[0132] Likewise, computer 60 may be comprised of multiple physically separate components (which may each have their own corresponding components).

[0133] Communication interface 604 may also include multiple sets of various illustrated components for different wireless technologies, such as, for example, GSM, WCDMA, LTE, NR, WiFi, or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chips or chipsets and other components within computer 600.

[0134] The processing circuitry 612 is configured to perform any determinations, calculations, or similar operations (e.g., certain obtaining operations) described herein as being provided by the computer 600. Such operations performed by the processing circuitry 612 may include processing information obtained by the processing circuitry 612 by, for example, converting the obtained information into other information, comparing the obtained information or the converted information with information stored in a network node, and / or performing one or more operations based on the obtained information or the converted information and making determinations as a result of the processing.

[0135] The processing circuit 612 may include a microprocessor, a controller, a microcontroller, a central processing unit, a digital signal processor, an application specific integrated circuit, a field programmable gate array, or a combination of one or more of any other suitable computing device, resource, or a combination of coded logic and / or software, hardware operable to provide computer 600 functionality alone or in combination with other computer 600 components (such as device readable media). For example, the processing circuit 612 may execute instructions stored in the device readable medium 615 or in a memory within the processing circuit 612. Such functionality may include providing any of the various wireless features, functions, or benefits discussed herein. In some embodiments, the processing circuit 612 may include a system on a chip (SOC).

[0136] In some embodiments, processing circuitry 612 may include one or more of radio frequency (RF) transceiver circuitry and baseband processing circuitry. In some embodiments, radio frequency (RF) transceiver circuitry and baseband processing circuitry may be on separate chips (or chipsets), boards, or units (such as a radio unit and a digital unit). In alternative embodiments, part or all of the RF transceiver circuitry and baseband processing circuitry may be on the same chip or chipset, board, or unit.

[0137] In some embodiments, some or all of the functionality described herein as being provided by the computer 600 may be performed by the processing circuit 612 executing instructions stored on the device readable medium 615 or a memory within the processing circuit 612. In alternative embodiments, some or all of the functionality may be provided by the processing circuit 612, for example, in a hardwired manner, without executing instructions stored on a separate or discrete device readable medium. In any of those embodiments, the processing circuit 612 can be configured to perform the described functionality, whether or not executing instructions stored on a device readable storage medium. The benefits provided by such functionality are not limited to the processing circuit 612 itself or other components of the computer 600, but are enjoyed by the computer 600 as a whole and / or by an end user.

[0138] The device-readable medium 615 may include any form of volatile or non-volatile computer-readable memory, including, but not limited to, permanent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (e.g., a hard disk), removable storage media (e.g., a flash drive, a compact disk (CD), or a digital video disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory device that stores information, data, and / or instructions that may be used by the processing circuit 612. The device-readable medium 615 may store any suitable instructions, data, or information, including computer programs, software, applications including one or more of logic, rules, codes, tables, etc., and / or other instructions that are executable by the processing circuit 612 and utilized by the computer 600. The device-readable medium QQ180 may be used to store any calculations performed by the processing circuit 612 and / or any data received via the interface 604. In some embodiments, the processing circuit 612 and the device-readable medium 615 may be considered to be integrated.

[0139] Interface 604 is used in wired or wireless communication of signaling and / or data between computer 600 and other nodes. Interface 604 may include (one or more) ports / (one or more) terminals for sending data to and receiving data from computer 600 via a wired connection, for example. Interface 604 also includes radio front-end circuitry, which may be coupled to an antenna, or may be part of an antenna in some embodiments. The radio front-end circuitry may include filters and amplifiers. The radio front-end circuitry may be connected to an antenna and / or processing circuitry 612.

[0140] Examples of computer 600 include, but are not limited to, smartphones, mobile phones, cellular phones, voice over IP (VoIP) phones, wireless local loop phones, desktop computers, personal digital assistants (PDAs), wireless cameras, game consoles or devices, music storage devices, playback devices, wearable terminal devices, wireless endpoints, mobile stations, tablets, laptop computers, laptop embedded devices (LEEs), laptop mounted devices (LMEs), smart devices, wireless client equipment (CPEs), vehicle-mounted wireless terminal devices, etc.

[0141] The communication interface 604 may encompass a wired and / or wireless network, such as a local area network (LAN), a wide area network (WAN), a computer network, a wireless network, a telecommunications network, another similar network, or any combination of these networks. The communication interface may be configured to include a receiver and a transmitter interface for communicating with one or more other devices over a communication network in accordance with one or more communication protocols (such as Ethernet, TCP / IP, SONET, ATM, optical, electrical, etc.). The transmitter and receiver interfaces may share circuit components, software, or firmware, or alternatively may be implemented separately.

[0142] In one embodiment, a computer is provided, the computer being configured to perform any of the method steps described herein.

[0143] In one embodiment, a computer program is provided, the computer program comprising computer executable instructions for causing the computer to perform any of the method steps described herein when the computer executable instructions are executed on a processing unit comprised in the computer.

[0144] In one embodiment, a computer program product is provided, the computer program product comprising a computer-readable storage medium having the above-mentioned computer program implemented therein.

[0145] In one embodiment, a carrier is provided, wherein the carrier contains the above-mentioned computer program. The carrier can be any one of an electronic signal, an optical signal, a radio signal or a computer-readable storage medium.

[0146] Finally, it should be understood that the present invention is not limited to the embodiments described above, but also relates to and incorporates all embodiments within the scope of the appended independent claims.

Claims

1. A computer-implemented method (300) for associating a proposed region representing an object in a video (V), the video (V) comprising consecutive frames (F n-1 , F n ), the method comprising: Get the previous frame (F n-1 )’s first target suggested area information, Determine the current frame (F n ), wherein the first and second target proposal region information at least indicate an appearance metric, a spatial position, and a detection probability of each target proposal region of the corresponding frame, By at least the previous frame (F n-1 ) is associated with the current frame (F n ) to associate objects in the video (V), wherein a distance metric (D) calculated based on the appearance metric, the spatial position and the detection probability is used. n,n-1 ) to associate the target suggestion area.

2. The method according to claim 1, wherein: Associating the target in the video further includes: Based on the distance metric (D n,n-1 ) will be the minimum distance of the previous frame (F n-1 ) is mapped to the current frame (F n ), Adjust the current frame (F n ), By determining the current frame (F) having a detection probability higher than a threshold (T) n ) to detect the current frame (F n )’s goal.

3. The method according to claim 2, wherein: The previous frame (F n-1 ) and the target proposal area of ​​the current frame (F n ) between target proposal regions (D n,n-1 ) is calculated as: D n,n-1 =d(X n ,X n-1 )+d(c n ,c n-1 ), where d(X n ,X n-1 ) is a combined spatial probability distance metric, and d(c n ,c n-1 ) is the appearance distance metric.

4. The method according to claim 3, wherein: The distance metric (D n,n-1 ) is calculated as a weighted sum of the combined spatial probability distance metric and the appearance distance metric.

5. The method according to any one of claims 3 to 4, wherein: The object proposal region is formed as a rectangle with two-dimensional center coordinates x, y, width w and height h, where the spatial probabilistic distance metric is calculated based on a vector formed by X=[x, y, w, h, p], where p is the detection probability of the object proposal region.

6. The method according to any one of claims 3 to 4, wherein: If the corresponding appearance distance metric is below the second threshold Then adjust the current frame (F n ) is the detection probability of each mapped object proposal region.

7. A computer, comprising: Processing circuit; and A memory storing instructions which, when executed by the processing circuit, cause the computer to be configured to perform the method according to any one of claims 1-6.

8. A computer program product storing computer executable instructions, which, when executed on a processing unit included in a computer, cause the computer to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer-executable instructions, which, when executed on a processing unit included in a computer, cause the computer to perform the method according to any one of claims 1 to 6.

Images