Method, device and electronic device for obtaining characteristic information of target object

Abstract

The embodiments of the present application disclose a method, device and electronic device for obtaining feature information of a target object. The method includes: the client obtains the size information of at least one available template, the size information includes the width information of the available template and the height information of the available template; obtains the search area corresponding to the target object to be tracked in the current frame image, creates a corresponding storage area for the search area, and saves the image information associated with the search area to the storage area; determines the target template, and scales the image information in the storage area according to the size information of the target template to generate a template image; extracts feature information related to the target object from the template image, so as to track the target object in the next frame image according to the feature information. Such a scheme reduces the time complexity of the algorithm while minimizing the impact on the tracking effect of the algorithm.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and electronic device for obtaining feature information of a target object. Background Art

[0002] Target tracking technology is a research hotspot in the field of computer vision. It is widely used in security (e.g., road traffic scenes), monitoring (e.g., face recognition scenes), inspection (e.g., drone tracking scenes), and other fields. It can track specific target objects in video sequences, associate target objects in the time and space domain, and obtain the motion trajectory of the target objects. In other words, after determining the position information of the target object to be tracked in the initial frame image of the video sequence, the target tracking technology can automatically locate the target object in the subsequent frame images and predict the position information of the target object in the subsequent frame images.

[0003] At present, the commonly used algorithms for target tracking are mainly based on the correlation filter target tracking algorithm. Under this algorithm, the search area where the target object is located can be determined based on the position information of the target object in the current frame image, and the image of the search area is scaled to generate a template image (patch), and then the feature information related to the target object is extracted from the template image, so as to perform feature matching in the next frame image, obtain the position information of the target object in the next frame image, and achieve the purpose of target tracking.

[0004] The correlation filter series of algorithms are developing rapidly. From MOSSE (Minimum Output Sum of Squared Error Filter) to CSK (Circulant Structure of Tracking-bydetection with Kernels), CN (Color Name), KCF (Kernelized Correlation Filters), DCF (Discriminative Correlation Filters) and other algorithms, the extracted feature information is becoming more and more abundant, and the positioning effect is getting better and better. However, the time complexity of the algorithm is gradually increasing, and the response speed is getting slower and slower.

[0005] How to balance the tracking effect of the target tracking algorithm and the time complexity of the algorithm has become a technical problem that needs to be solved by technical personnel in this field. Summary of the invention

[0006] The present application provides a method, device and electronic device for obtaining feature information of a target object, which effectively reduces the time complexity of the algorithm while minimizing the impact on the tracking effect of the algorithm.

[0007] This application provides the following solutions:

[0008] A method for obtaining feature information of a target object, comprising:

[0009] The client obtains size information of at least one available template, where the size information includes width information of the available template and height information of the available template;

[0010] Obtaining a search area corresponding to the target object to be tracked in the current frame image, creating a corresponding storage area for the search area, and saving image information associated with the search area to the storage area;

[0011] Determine a target template from the available templates, and perform scaling processing on the image information in the storage area according to the size information of the target template to generate a template image;

[0012] Feature information related to the target object is extracted from the template image so as to track the target object in the next frame image according to the feature information.

[0013] A target tracking processing method, comprising:

[0014] The client obtains size information of at least one available template, where the size information includes width information of the available template and height information of the available template;

[0015] If a video sequence related to a vehicle traveling on a road is obtained, a target object to be tracked is determined from the video sequence;

[0016] A target template is determined from the available templates, feature information of the target object is obtained using the target template, and tracking processing is performed on the target object.

[0017] A device for obtaining characteristic information of a target object, applied to a client, comprising:

[0018] A size information obtaining unit, used to obtain size information of at least one available template, wherein the size information includes width information of the available template and height information of the available template;

[0019] A search area obtaining unit, used to obtain a search area corresponding to the target object to be tracked in the current frame image;

[0020] An image information storage unit, used to create a corresponding storage area for the search area, and save the image information associated with the search area to the storage area;

[0021] A target template determining unit, configured to determine a target template from the available templates;

[0022] A template image generating unit, used for performing scaling processing on the image information in the storage area according to the size information of the target template to generate a template image;

[0023] The feature information extraction unit is used to extract feature information related to the target object from the template image, so as to track the target object in the next frame image according to the feature information.

[0024] A target tracking processing device, applied to a client, comprising:

[0025] A size information obtaining unit, used to obtain size information of at least one available template, wherein the size information includes width information of the available template and height information of the available template;

[0026] a target object determination unit, configured to determine a target object to be tracked from a video sequence when a video sequence related to a vehicle traveling on a road is obtained;

[0027] The tracking processing unit is used to determine a target template from the available templates, obtain feature information of the target object using the target template, and perform tracking processing on the target object.

[0028] An electronic device, comprising:

[0029] one or more processors; and

[0030] A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the following operations are performed:

[0031] Obtaining size information of at least one available template, wherein the size information includes width information of the available template and height information of the available template;

[0032] Obtaining a search area corresponding to the target object to be tracked in the current frame image, creating a corresponding storage area for the search area, and saving image information associated with the search area to the storage area;

[0033] Determine a target template, and perform scaling processing on the image information in the storage area according to the size information of the target template to generate a template image;

[0034] Feature information related to the target object is extracted from the template image so as to track the target object in the next frame image according to the feature information.

[0035] An electronic device, comprising:

[0036] one or more processors; and

[0037] A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the following operations are performed:

[0038] Obtaining size information of at least one available template, wherein the size information includes width information of the available template and height information of the available template;

[0039] If a video sequence related to a vehicle traveling on a road is obtained, a target object to be tracked is determined from the video sequence;

[0040] A target template is determined from the available templates, feature information of the target object is obtained using the target template, and tracking processing is performed on the target object.

[0041] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0042] In an embodiment of the present application, the size information of the available template obtained by the client may include the width information and height information of the available template. In this way, after obtaining the search area corresponding to the target object in the current frame image, the target template determined from the available templates can be used to scale the search area to generate a template image of a fixed size, and then extract feature information from it. In other words, the size of the generated template image can be limited by the size information of the target template, which helps to reduce the time consumed by the feature information process, thereby reducing the time complexity of the target tracking algorithm. In particular, when the search area is elongated, the target template provided by the embodiment of the present application can be used to avoid dividing a large number of cells in the generated template image due to the excessive size of the long side of the search area, resulting in a long time for feature information extraction. In addition, for search areas of different sizes, template images of the same size can be generated by the same target template, which also helps to improve the controllability of the algorithm's time complexity.

[0043] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 is a flow chart of a method provided by an embodiment of the present application;

[0046] Figure 2 is a schematic diagram of a bilinear difference provided in an embodiment of the present application;

[0047] Figure 3 It is a schematic diagram of extracting FHOG feature information provided by an embodiment of the present application;

[0048] Figure 4 It is a comparison chart of the time complexity of the embodiment of the present application and the prior art;

[0049] Figure 5 is a flow chart of another method provided in an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of a device provided in an embodiment of the present application;

[0051] Figure 7 is a schematic diagram of another device provided in an embodiment of the present application;

[0052] Figure 8 It is a schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0054] In order to facilitate understanding of the specific implementation scheme provided by the embodiment of the present application, the specific scenario of this scheme is briefly introduced below.

[0055] Through target tracking technology, the target object in the previous frame image can be automatically located in the current frame image, and then the target objects in the previous and next frame images can be associated.

[0056] Specifically, after obtaining the target video sequence to be processed, the position information of the target object can be determined from the initial frame image of the target video sequence, and then the target object can be located and tracked in subsequent frame images by a target tracking algorithm (for example, a target tracking algorithm based on correlation filtering). Taking the target video sequence as a road traffic video as an example, the target object can be a specific object such as a vehicle or a pedestrian appearing in the target video sequence.

[0057] As an example, the location information of the target object can be determined in the initial frame image through the target frame, that is, the area enclosed by the target frame is the location of the target object. For example, the location information of the target object can be manually selected by a user associated with the client from the initial frame image.

[0058] Alternatively, the position information of the target object can be automatically detected in the initial frame image by deep learning. Taking the vehicle detection model trained in advance by deep learning as an example, if the initial frame image is used as input and a unique detection result is output after processing by the vehicle detection model, the vehicle represented by the unique detection result can be determined as the target object, and the position determined by the unique detection result can be used as the position information of the target object for tracking processing; if at least two detection results are output after processing by the vehicle detection model, the vehicle information represented by the two detection results can be displayed through the client so that the user can determine the target object to be tracked.

[0059] It can be understood that the initial frame image can be any frame image in the target video sequence, for example, the first frame image of the target video sequence is determined as the initial frame image. After determining the position information of the target object from the initial frame image according to the implementation method described above, the search area where the target object is located can be obtained according to the position information, and then the image of the search area is scaled to generate a template image, and then feature information related to the target object is extracted from the template image, so as to locate and track the target object in subsequent frame images by feature matching.

[0060] At present, when generating a template image, the prior art usually pre-sets the size information of the short side of the template image, and then compares the height information and width information of the search area to determine the shorter side from the two, and then performs scaling based on the shorter side.

[0061] Take the template size information of 64 as an example, that is, the size of the short side of the template image generated by the scaling process is 64 pixels. If the size information of the search area is 500x100, where the height is the shorter side, the scaling process can be performed based on the height. Specifically, the image information in the height direction of the search area can be downsampled first, the height of the search area can be reduced to 64, and then the image information in the width direction of the search area can be downsampled according to the aspect ratio of the search area, and the width of the search area can be proportionally reduced to 320. The size information of the template image generated by the scaling process is 320x64. If the size information of the search area is 128×200, the image information in the width direction can be downsampled to 64 first, and then the image information in the height direction can be proportionally reduced to 100. The size information of the template image generated by the scaling process is 64x100.

[0062] When extracting feature information related to the target object from the template image, the template image can be first divided into a number of local area blocks of fixed size cells, and then feature information is extracted based on the cells. The size of the cell can be 4×4, 8×8, etc., and this embodiment of the application does not make specific limitations on this.

[0063] Taking the cell size of 4×4 as an example, in the two template images generated by the scaling process above, the 320x64 template image can be divided into 80×16 cells, including 78×14 valid cells from which feature information can be extracted (see the introduction below for details, which will not be described in detail here), that is, the amount of feature information contained in each feature dimension is 78×14; the 64x100 template image can be divided into 16×25 cells, including 14×23 valid cells from which feature information can be extracted, that is, the amount of feature information contained in each feature dimension is 14×23. In other words, the size information of the template image and the size information of the cell determine the number of cells included in the template image, and when the cell size is fixed, the larger the size of the template image, the more cells are divided, the greater the amount of feature information contained in each feature dimension, and the longer the response time of the algorithm.

[0064] In summary, the prior art generates a template image by presetting the short side size information of the template image and scaling the image in the search area in equal proportion, which may increase the time complexity of the algorithm and consume a lot of computing power, especially when the search area is elongated. The impact on the complexity of the algorithm is very obvious. In addition, for search areas of different sizes, template images of different sizes may be generated, and the time consumed for feature information extraction of template images of different sizes will inevitably be different, resulting in uncontrollable time complexity of the algorithm.

[0065] In order to shorten the response time of the algorithm and reduce the time complexity of the algorithm, an embodiment of the present application provides a tool for obtaining feature information of a target object, which may include a client deployed on a terminal device associated with a user. The client may obtain the size information of at least one available template, and the size information of each available template may include the width information and height information of the available template. In this way, when the client obtains the search area corresponding to the target object, the target template may be determined from the available templates, and the search area may be scaled according to the size information of the target template to generate a template image of a fixed size, and then feature information may be extracted from it. In other words, the size information of the target template may be used to limit the size of the generated template image, so as to avoid the problem that when the search area is elongated, a large number of cells are divided in the template image due to the excessive size of the long side, resulting in the feature information extraction being time-consuming and consuming a lot of computing power.

[0066] Taking the target template size information of 64×64 as an example, for the search area of ​​500x100 and the search area of ​​128×200 mentioned above, the size information of the template image generated by the scaling process is 64x64. If the size of the cell is 4×4, 16×16 cells can be divided in the template image of the embodiment of the present application, including 14×14 valid cells from which feature information can be extracted. Compared with the prior art, the number of cells is greatly reduced, which can effectively reduce the response time of the algorithm. In addition, for search areas of different sizes, template images of the same size can be generated by the same target template, which also helps to improve the controllability of the algorithm's time complexity.

[0067] Combine the following Figure 1 The flowchart shown explains the implementation process of the embodiment of the present application.

[0068] Example 1

[0069] S101: The client obtains size information of at least one available template, where the size information includes width information of the available template and height information of the available template.

[0070] After obtaining the search area corresponding to the target object, it is usually necessary to scale the image of the search area to generate a template image. This is mainly to reduce the amount of feature information contained in each feature dimension and improve the processing efficiency of the algorithm. As mentioned above, in some cases, for example, when the search area is elongated, if the aspect ratio of the search area is kept unchanged for scaling, it may not be possible to maximize the reduction of the amount of feature information contained in each feature dimension. For example, for the template image of size 320x64 mentioned above, the feature information that can be extracted is 78x14xd, where d is the number of feature dimensions, and the amount of feature information contained in each feature dimension is 78×14. From this example, it can be seen that the longer side of the template image, that is, the amount of feature information contained in the width direction is still large, and there is the possibility of further optimization.

[0071] In order to minimize the amount of feature information contained in each feature dimension, the embodiment of the present application can pre-configure the size information of the available template in the width and height directions to avoid proportional scaling of the image in the search area resulting in a situation where the size of one side of the template image is too large.

[0072] In one implementation, the shape of the available template can be a square. For example, the size information of the available template is N×N, that is, the available template includes N pixels in both the width direction and the height direction. The value of N can be set according to the actual application requirements to avoid the value of N being too large, resulting in too much feature information contained in each feature dimension, and to avoid the value of N being too small, resulting in a cosine window in which most parameters degenerate to zero. As an example, the value range of N can be 64 to 128 pixels, and the embodiments of the present application may not make specific limitations on this. Among them, the cosine window is mainly used to smooth the extracted feature information. The specific implementation process can refer to the relevant technology and will not be described in detail here.

[0073] Alternatively, in another implementation, the shape of the available template may be a rectangle. For example, the size information of the available template is n×M or M×n, where M<n, and the value of M is the minimum value to ensure the effectiveness of the cosine window function. As an example, the value of M may be 16 or 32, and the embodiment of the present application may not be specifically limited to this.

[0074] For the above example, the information of available templates obtained by the client may be shown in Table 1 below.

[0075] Table 1

[0076] Number information of available templates Size information for available templates 1 N×N 2 n×M 3 M×n

[0077] In the embodiment of the present application, the client can obtain the size information of the available templates in a variety of ways. For example, the size information of the available templates can be stored in a server deployed on a cloud server, and the client can access the server to obtain the size information of at least one available template from the server. Alternatively, the size information of the available templates can be pre-packaged in the implementation code of the client, and the client can run the implementation code to obtain the size information of at least one available template by locally reading.

[0078] In addition, in order to improve the flexibility of feature information extraction in the embodiment of the present application, the user can customize the size information of the available template according to the actual application requirements. Specifically, the client can provide an operation option for obtaining the size information of the newly added available template, and save the size information of the newly added available template locally after obtaining it through the operation option, or submit the user-defined size information of the newly added available template to the server for saving, and the embodiment of the present application does not make specific limitations on this.

[0079] S102: Obtaining a search area corresponding to the target object to be tracked in the current frame image.

[0080] When the client extracts feature information in the current frame image, it can first determine the position information of the target object in the current frame image, and then obtain the search area corresponding to the target object according to the position information.

[0081] If the current frame image is the initial frame image, the client can determine the position information of the target object in the current frame image by manual selection or deep learning as described above. If the current frame image is a subsequent frame image, considering that the displacement of the target object in the two adjacent frames is usually not too large, the client can determine the position information obtained by locating the target object in the previous frame image as the position information of the target object in the current frame image.

[0082] As an example, if the location information of the target object is determined by the target frame, a region larger than the target frame size can be determined as the search region corresponding to the target object with the center of the target frame as the center, and the search region can also be called ROI (region of interest). For example, if the size information of the target frame is 60×60, if a region 2.5 times the size of the target frame is determined as the search region corresponding to the target object, the size information of the search region is 150×150.

[0083] In actual application, if the search area exceeds the boundary of the current frame image, resulting in the search area including some blank areas, the image information adjacent to the blank area in the current frame image can be used to fill the boundaries of the image in the search area to avoid affecting the accuracy of locating the target object due to the failure to extract effective feature information from the blank area.

[0084] It is understandable that for the same number of filled rows or columns, the larger the size of the search area, the longer it takes to fill the borders. In view of this, the embodiment of the present application can provide the following preferred solution: when the search area exceeds the boundary of the current frame image, the current frame image can be downsampled first to obtain a processed current frame image of relatively smaller size. In this way, the size of the new search area in the processed current frame image will also become smaller, and the border of the new search area of ​​smaller size is filled, which helps to reduce the time complexity of the algorithm.

[0085] Specifically, when it is determined that the search area exceeds the boundary of the current frame image, the current frame image can be downsampled; or, when it is determined that the search area exceeds the boundary of the current frame image, the number of rows (which can be recorded as dx) and the number of columns (which can be recorded as dy) for filling the boundary of the search area can be obtained, and when the number of rows and the number of columns meet the first preset condition, the current frame image can be downsampled.

[0086] The first preset condition is described below with an example.

[0087] In one implementation, the first preset condition may be that the maximum value of the row number information and the column number information exceeds the first threshold value T1, that is, max(dx, dy)>T1. As an example, the first threshold value T1 may be determined by the minimum value of the width information W and the height information H of the current frame image, and the first coefficient k1, for example, T1=k1×min(W, H), where the value range of k1 may be 0.1 to 0.5. In this implementation, the first downsampling multiple S1 for downsampling the current frame image may be determined based on the maximum value of the row number information and the column number information, and the first threshold value, for example, S1=max(dx, dy) / T1.

[0088] Alternatively, in another implementation, the first preset condition may be that the minimum value of the row number information and the column number information exceeds the second threshold value T2, that is, min(dx, dy)>T2. As an example, the second threshold value T2 may be determined by the minimum value of the width information and the height information of the current frame image, and the second coefficient k2, for example, T2=k2×min(W, H), where the value range of k2 may be 0.1 to 0.2. In this implementation, the second downsampling multiple S2 for downsampling the current frame image may be determined based on the maximum value of the row number information and the column number information, and the second threshold value, for example, S2=max(dx, dy) / T2.

[0089] The implementation process of this preferred solution is described below with examples.

[0090] If the size information of the current frame image is 1280×720; the position information of the target frame is (x1, y1), which can be specifically reflected as the coordinate information of the upper left corner of the target frame, and the size information of the target frame is (w1, h1); the position information of the search area determined according to the target frame is (x2, y2), which can be reflected as the coordinates of the upper left corner of the search area, and the size information of the search area is (w2, h2).

[0091] The processing process can be reflected as follows:

[0092] First, the number of rows and columns to be filled is obtained based on the location information and size information of the search area. Specifically,

[0093] The number of columns to be filled is dy = max(-x2, 0) + max(x2 + w2 - W, 0)

[0094] The number of rows to be filled is dx = max(-y2, 0) + max(y2 + h2 - H, 0)

[0095] In this example, the coordinate x0 of the left boundary of the current frame image on the x-axis can be set to 0. The value of max(-x2, 0) can be used to determine whether the left side of the search area exceeds the left boundary of the current frame image. If max(-x2, 0) = 0, it means that it does not exceed the left boundary, and the number of columns that need to be filled on the left side dy1 = 0; otherwise, it means that it exceeds the left boundary, and dy1 = x2; the value of max(x2+w2-W, 0) can be used to determine whether the right side of the search area exceeds the right boundary of the current frame image. If max(x2+w2-W, 0) = 0, it means that it does not exceed the right boundary, and the number of columns that need to be filled on the right side dy2 = 0; otherwise, it means that it exceeds the right boundary, and dy2 = x2+w2-W.

[0096] In addition, the coordinate y0 of the upper boundary of the current frame image on the y-axis can be set to 0 (where downward is the positive direction of the y-axis). Max(-y2, 0) can be used to determine whether the upper side of the search area exceeds the upper boundary of the current frame image, and the number of rows dx1 that need to be filled; max(y2+h2-H, 0) can be used to determine whether the lower side of the search area exceeds the lower boundary of the current frame image, and the number of rows dx2 that need to be filled.

[0097] Next, the first threshold T1 is determined according to the size information of the current frame image. If max(dx, dy)>T1, it can be determined that the current frame image can be downsampled first, and the first downsampling multiple S1=max(dx, dy) / T1. It should be noted that in this example, the height information of the current frame image is smaller than the width information, so T1 can be determined according to the height information of the current frame image. If k1=0.1, then T1=0.1*720=72.

[0098] Finally, the current frame image is downsampled according to S1 to obtain the processed current frame image. At the same time, the size information and position information of the search area can be adjusted according to S1 to obtain the size information (w2 / S1, h2 / S1) and position information (x2 / S1, y2 / S1) of the new search area. In the subsequent processing, the blank area in the new search area can be border filled. Compared with directly filling the border of the search area of ​​size (w2, h2), such processing helps to reduce the time consumption of the border filling process, thereby reducing the time complexity of the algorithm.

[0099] In actual application, the client can obtain a unique downsampling factor through an implementation method as shown in the example above to downsample the current frame image; or, the client can obtain the downsampling factors S1 and S2 respectively through the two implementation methods introduced above, and select the largest downsampling factor max(S1, S2) therefrom to downsample the current frame image. The embodiments of the present application do not make specific limitations on this.

[0100] S103: Create a corresponding storage area for the search area, and save image information associated with the search area to the storage area.

[0101] In an embodiment of the present application, a corresponding storage area can be created for the search area, and image information associated with the search area can be copied to the storage area before scaling, thereby preventing the image of the search area from being directly scaled in the current frame image, which would affect the current frame image.

[0102] As an example, the association relationship between the storage area created by the client and the image frame to which the search area belongs may be shown in Table 2 below.

[0103] Table 2

[0104] Image frame identification information Storage area identification information Image frame 1 Storage Area 1 Image frame 2 Storage Area 2 …… ……

[0105] It is understandable that the larger the size of the search area, the more image information needs to be copied to the storage area, the longer the copying process takes, and the time complexity of the algorithm will also increase. In view of this, the embodiment of the present application can provide the following preferred solution: when the size information of the search area meets the second preset condition, that is, when the size of the search area is too large, the current frame image can be downsampled first to obtain a processed current frame image with a relatively small size. In this way, the size of the new search area in the processed current frame image will also become smaller, and the image information associated with the new search area with a smaller size is copied, which helps to reduce the time complexity of the algorithm.

[0106] The second preset condition is described below with an example.

[0107] In one implementation, the second preset condition may be that the maximum value of the height information h2 and the width information w2 of the search area exceeds the third threshold value T3, that is, max(w2, h2)>T3. As an example, the third threshold value may be determined by the minimum value of the width information and the height information of the current frame image, and the third coefficient k3, for example, T3=k3×min(W, H), where the value range of k3 may be 0.5 to 1.0. In this implementation, the third downsampling multiple S3 for downsampling the current frame image may be determined based on the third coefficient, for example, S3=1 / k3.

[0108] Alternatively, in another implementation, the second preset condition may be that the minimum value of the distance between the boundary of the search area and the corresponding boundary of the current frame image is less than a fourth threshold value T4. In this implementation, the fourth downsampling multiple S4 for downsampling the current frame image may be embodied as S4=min(h2, w2) / TW, where TW is the preset side length of the new search area in the current frame image after downsampling, and TW is not greater than the length of the shortest side of the target template. For example, if the size of the target template is 64×64, the value of TW may be 64.

[0109] Among them, the distance between the boundary of the search area and the corresponding boundary of the current frame image (which can be reflected as the absolute value of the distance) may include: the distance value △x1 from the left boundary of the search area to the left boundary of the current frame image, the distance value △x2 from the right boundary of the search area to the right boundary of the current frame image, the distance value △y1 from the upper boundary of the search area to the upper boundary of the current frame image, and the distance value △y2 from the lower boundary of the search area to the lower boundary of the current frame image. If min(△x1, △x2, △y1, △y2)<T4, the current frame image can be downsampled first according to this preferred scheme.

[0110] For the fourth threshold value T4, if the ratio between the size information of the current frame image and the size information of the target object remains unchanged, T4 can be determined based on the size information of the current frame image. For example, the size information of the current frame image is 200×200, and the size information of the vehicle as the target object is 10×10. If the ratio between the two size information remains unchanged, 10% of the size of the current frame image can be determined as the fourth threshold value, that is, T4=20. In addition, if the ratio between the size information of the current frame image and the size information of the target object changes, the fourth threshold value T4 can be obtained based on the size information of the current frame image. Specifically, the ratio between the height information of the current frame image and the height information of the search area can be obtained. And the ratio between the width information of the current frame image and the width information of the search area The side length of the current frame image corresponding to the maximum ratio is determined as the basic side length. For example, if The width of the current frame image can be determined as the basic side length; if Then the height of the current frame image can be determined as the basic side length. Finally, the fourth threshold is obtained according to the length information S of the basic side length and the fourth coefficient k4, for example, T4=k4×S, where the value range of k4 can be 0.05-0.1.

[0111] The implementation process of this preferred solution is described below with examples.

[0112] If the size information of the current frame image is 1280×720, because the height information of the current frame image is smaller than the width information, T3 can be determined according to the height information H. If k3=0.5, then T3=0.5*720=360. The size information of the search area is obtained. If max(w2, h2)>T3, the current frame image can be downsampled first, and the third downsampling multiple S3=1 / k3=2.

[0113] At the same time, the client can also adjust the size information and position information of the search area according to S3 to obtain the new size information (w2 / S3, h2 / S3) and position information (x2 / S3, y2 / S3) of the new search area. In the subsequent processing, the image information associated with the new search area can be copied to the storage area. Compared with directly copying the image information of the size (w2, h2), such processing helps to reduce the time consumption of the copy process, thereby reducing the time complexity of the algorithm.

[0114] In actual application, the client can obtain a unique downsampling factor through an implementation method as shown in the example above to downsample the current frame image; or, the client can obtain the downsampling factors S3 and S4 respectively through the two implementation methods introduced above, and select the largest downsampling factor max(S3, S4) from them to downsample the current frame image. The embodiments of the present application do not make specific limitations on this.

[0115] S104: Determine a target template, and perform scaling processing on the image information in the storage area according to the size information of the target template to generate a template image.

[0116] In one implementation, the target template may be set by default, for example, the N×N available templates may be set by default as the target template. As an example, the implementation code of the client may include a line of code for configuring the target template by default, so that when the client performs target tracking, the target template can be determined by running the line of code. Or, as another example, the user may select a target template from the available templates for default setting according to actual application requirements. Specifically, the client may display the size information of the available templates, and when the target template is selected, the target template may be set by default. In this way, when the client performs target tracking, the template with the default setting may be determined as the target template for template image generation.

[0117] Alternatively, in another implementation, the target template can be determined from the available templates according to the aspect ratio of the search area in the initial frame image. Specifically, the target aspect ratio of the search area corresponding to the target object determined in the initial frame image can be obtained, and the available template close to the target aspect ratio can be determined as the target template. For example, the client obtains 3 available templates: N×N (i.e., the height information and the width information are the same), n×M (i.e., the width information is greater than the height information), and M×n (i.e., the width information is less than the height information). If the size information of the search area is 500×100, i.e., the width information is greater than the height information, then n×M can be determined as the target template; if the size information of the search area is 128×200, then M×n can be determined as the target template; if the size information of the search area is 128×128, then N×N can be determined as the target template.

[0118] In addition, if at least two available templates with different aspect ratios are obtained for the same shape, for example, the size information of available template 1 is 64×16, and the size information of available template 2 is 32×16, both of them belong to available templates of n×M shape, but with different aspect ratios. If the size information of the search area is 500×100, that is, the target aspect ratio is 5:1, combined with the aspect ratio of available template 1 of 4:1 and the aspect ratio of available template 2 of 2:1, the available template 1 with the aspect ratio closest to the target aspect ratio can be determined as the target template, which not only helps to reduce the time complexity of the algorithm, but also minimizes the deformation of the target object.

[0119] It can be understood that after the target template is determined through the search area in the initial frame image, when performing target tracking processing on subsequent frame images, the determined target template can be directly used to generate a template image.

[0120] Taking the size information of the search area as 500×100 as an example, if the size information of the target template is 64×64, the image information of the search area in the width direction can be downsampled to 64, and the image information of the search area in the height direction can be downsampled to 64. The specific process can be reflected as follows:

[0121] First, according to the size information of the search area and the size information of the target template, the scaling ratio scaleX=w0 / w2=500 / 64 in the width direction and the scaling ratio scaleY=h0 / h2=100 / 64 in the height direction are calculated respectively. Among them, w0 is the width information of the target template, h0 is the height information of the target template, w2 is the width information of the search area, and h2 is the height information of the search area.

[0122] It can be understood that if a new search area is obtained according to the preferred scheme introduced in S102 and S103, the scaling ratios in the width direction and the height direction can be calculated respectively according to the size information of the new search area and the size information of the target template. The specific implementation process can be referred to the introduction above and no examples will be given here.

[0123] Secondly, the image information in the width direction of the search area is scaled according to scaleX, and the image information in the height direction of the search area is scaled according to scaleY to obtain a template image.

[0124] As an example, the image information can be scaled by bilinear interpolation. Figure 2 The schematic diagram shown explains the implementation process of bilinear interpolation.

[0125] 1. Calculate the pixel p in the template image according to the scaling ratios scaleX and scaleY t (x t ,y t ) corresponds to the position p in the search area r (x r ,y r ), which can be specifically expressed as the following formula 1:

[0126]

[0127] 2. Determine p in the image of the search area r Four adjacent pixels (Q 12 , Q 22 , Q 11 , Q 21 ), and calculate p according to the pixel values ​​of the four pixels r The pixel value of the template image p t The pixel value can be specifically expressed as the following formula 2 and formula 3:

[0128]

[0129]

[0130] According to the above process, the pixel value of each pixel in the template image can be calculated, and the process of scaling the image in the search area to generate the template image can be completed.

[0131] In the above scaling process, if the size of the search area is smaller than the size of the target template, the template image can be generated by upsampling the image of the search area; if the size of the search area is larger than the size of the target template, the template image can be generated by downsampling the image of the search area. The image of the search area can be scaled according to actual application requirements.

[0132] It should be noted that the target template in the embodiment of the present application maintains the preset size information unchanged, and the target object in the generated template image may be deformed. Considering that when tracking the target in the previous and next frame images, the size information of the target template is fixed and the size information of the search area is fixed, resulting in the same proportion of deformation each time, so the deformation has little effect on the extraction of feature information. In addition, if the target object is a rigid body, such as a vehicle, the possibility of deformation causing the weakening of the characteristic information is also low. Therefore, even if the processing process of the embodiment of the present application may cause the target object to deform, the impact on the positioning effect of the target tracking algorithm is small.

[0133] S105: Extracting feature information related to the target object from the template image, so as to track the target object in the next frame image according to the feature information.

[0134] The following uses the KCF algorithm as an example to explain the implementation process of extracting feature information from a template image.

[0135] When the KCF algorithm is implemented, the following two types of feature information can be extracted from the template image:

[0136] 1. Advanced HOG[4] based on HOG (Histogram of Oriented Gradient), which can be denoted as FHOG.

[0137] In an embodiment of the present application, HOG features can be obtained by calculating and counting the cell gradient direction histogram. When extracting HOG feature information, 0 degrees to 360 degrees (corresponding to the extraction of signed HOG feature information) or 0 degrees to 180 degrees (corresponding to the extraction of unsigned HOG feature information) can be divided into several intervals, and each interval can be called a bin. For example, every 20 degrees is divided into a bin, then the signed HOG feature information can correspond to 18 bins, that is, it has 18 feature dimensions, and the unsigned HOG feature information can correspond to 9 bins, that is, it has 9 feature dimensions.

[0138] When extracting feature information, we can count the number of different bins corresponding to the gradient direction angle value in each cell to obtain the signed HOG feature information of the cell; we can also count the number of different bins corresponding to the absolute value of the gradient direction angle value in each cell to obtain the unsigned HOG feature information of the cell.

[0139] Combine the following Figure 3 The schematic diagram shown explains the implementation process of extracting FHOG feature information.

[0140] First, each cell can form a block with the surrounding 8 cells, that is, a block can include 9 cells. For cell0 at the center of the block, it can form a small block1 including 4 cells with cell8, cell1 and cell2, form a small block2 including 4 cells with cell2, cell3 and cell4, form a small block3 including 4 cells with cell4, cell5 and cell6, and form a small block4 including 4 cells with cell6, cell7 and cell8. In other words, one block can determine four small blocks.

[0141] Secondly, for each small block, the signed HOG feature information and unsigned HOG feature information of each cell included in it can be obtained, and the HOG feature information of each cell can be combined, and then the HOG features of the four cells in the small block are normalized, and truncated according to the preset threshold to obtain the HOG feature information of each small block. Among them, the truncation process means that if the proportion of the statistical results corresponding to a bin in the HOG feature information exceeds the threshold, for example, the threshold is 0.2, then the proportion of the statistical results corresponding to the bin can be determined to be 0.2.

[0142] Figure 3 In the example given, each small block can obtain 18-dimensional signed HOG feature information and 9-dimensional unsigned HOG feature information. By averaging the signed HOG feature information and unsigned HOG feature information of the four small blocks, the feature information of the first 27 dimensions in the FHOG feature information can be obtained.

[0143] Finally, by averaging the 27-dimensional HOG feature information corresponding to each small block, we can obtain the feature information of the last 4 dimensions in the FHOG feature information. In other words, 31-dimensional FHOG feature information can be extracted from the template image.

[0144] 2. Quantify the template image in the Lab color space to obtain Lab feature information.

[0145] The Lab color model is a color system based on physiological characteristics, consisting of three elements: brightness (L) and a and b related to color. Among them, the L component represents brightness (Luminosity), the a component represents the range from red to green, and the b component represents the range from yellow to blue. The value range of L can be from 0 to 100, and when L = 50, it is equivalent to 50% black; the value range of a and b can be from +127 to -128, among which a = +127 represents red, and when the value of a gradually transitions to -128, it becomes green; when b = +127 represents yellow, and when the value of b gradually transitions to -128, it becomes blue. All colors can be composed of the interactive changes of these three components.

[0146] The process of extracting Lab feature information is explained below.

[0147] First, the natural images are clustered in the Lab color space to obtain 15 centroids, each of which corresponds to a set of (L, a, b) values.

[0148] Secondly, the pixels in each cell are quantized. For example, the distance from each pixel in the cell to the 15 centroids can be calculated, and the pixels can be divided into groups corresponding to the nearest centroid, so as to obtain the quantization results of all pixels in the cell corresponding to the 15 centroids as the 15-dimensional Lab feature information of the cell.

[0149] Finally, the Lab feature information of all cells is spliced ​​to obtain the Lab feature information of the template image.

[0150] As can be seen from the above introduction, the number of cells can be determined by the size information of the template image and the size information of the cell. For example, if the size information of the template image is 64x64 and the size information of the cell is 4x4, the template image can be divided into 16x16 cells. According to the implementation method of FHOG feature extraction, the cells at the outermost circle of the template image cannot be used for feature information extraction. Therefore, the template image can be divided into 14x14 valid cells. In other words, the FHOG feature information extracted from the template image is 14x14x31, and the Lab feature information is 14x14x15. If the FHOG feature information and the Lab feature information are directly concatenated according to the channel, in the example of the template image mentioned above, 14x14x46 feature information can be obtained, where the number of feature dimensions is 46, and the amount of feature information included in each feature dimension is 14x14.

[0151] In summary, feature information related to the target object can be extracted, and then the target object can be automatically located in the next frame of the image by feature matching. Specifically, a detection base sample can be obtained in the next frame of the image, and then multiple offset samples can be obtained based on the detection base sample. Then, a pre-trained classifier is used to perform feature matching on the detection base sample and the offset sample, and the sample with the highest confidence is determined as the target sample. Then, the position information of the target object in the next frame of the image is obtained by the offset of the target sample relative to the detection base sample. The specific matching process can be found in the relevant technology, which will not be described in detail here.

[0152] When using the feature information in the current frame image and the feature information of the sample in the next frame image for feature matching, it is necessary to obtain the relevant responses of the two in each feature dimension respectively, and accumulate the relevant response results of d feature dimensions as the response result of feature matching. Among them, the size of the feature information included in each feature dimension determines the complexity of the relevant response operation, and the number of feature dimensions determines how many times the relevant response operation needs to be run. In other words, the number of feature dimensions and the amount of feature information included in each feature dimension will affect the time complexity of the algorithm.

[0153] The above solutions, such as limiting the size of the template image by the size information of the target template, downsampling the current frame image first and then filling the boundaries of the search area, downsampling the current frame image first and then copying the image information associated with the search area, etc., all help to reduce the amount of feature information included in each feature dimension, thereby reducing the time complexity of the algorithm.

[0154] In addition, the embodiments of the present application can also provide the following two preferred solutions, which help to reduce the number of feature dimensions and thus reduce the time complexity of the algorithm.

[0155] Preferred solution 1 can reduce the number of dimensions of FHOG feature information.

[0156] As can be seen from the above introduction, FHOG feature information can include signed HOG feature information and unsigned HOG feature information, wherein the signed HOG feature information can represent the unsigned HOG feature information to a certain extent, that is, there may be redundancy in the FHOG feature information. Therefore, in this preferred solution, the code line for extracting unsigned HOG feature information can be deleted in the implementation code for extracting FHOG features. In this way, the unsigned HOG feature information can be removed when extracting FHOG feature information (that is, the unsigned HOG feature information can be removed). Figure 3 That is, the FHOG feature information may include: 18-dimensional signed HOG feature information, and 4-dimensional feature information obtained by averaging the 18-dimensional signed HOG feature information, that is, the extracted FHOG feature information may include 22 feature dimensions.

[0157] The second preferred solution can reduce the number of dimensions of Lab feature information.

[0158] As can be seen from the above introduction, Lab feature information of several dimensions can be obtained by performing quantization processing based on several centroids. Therefore, this preferred solution can reduce the number of dimensions of Lab feature information by reducing the number of centroids.

[0159] If 15 centroids are clustered to obtain A new centroids, the pixels in the template image can be re-quantized according to the new centroids to obtain A-dimensional Lab feature information. Where A is less than 15, the value of A can be determined according to actual application requirements to avoid A being too large, resulting in too many feature dimensions, and A being too small, resulting in a weak effect of Lab feature information in the image features. As an example, the value of A can be 5, that is, the number of dimensions of Lab feature information can be reduced to 5 dimensions.

[0160] If feature information is extracted according to the above two preferred schemes, for the template image of size 64×64, the FHOG feature information that can be extracted is 14x14x22, the Lab feature information is 14x14x5, and after channel concatenation, 14x14x27 feature information can be obtained. Compared with the feature information of size 14x14x46 mentioned above, the time complexity of the algorithm can be effectively reduced.

[0161] In addition, considering that Lab color features are easily affected by factors such as lighting changes, appropriately reducing the dimension of Lab feature information can also help maintain the stability of the overall feature information.

[0162] The beneficial effects of the embodiments of the present application are illustrated below with reference to specific examples.

[0163] 206 experimental video sequences were obtained, including real road traffic scenes shot at different locations, covering sunny, rainy, daytime, night and other scenes, and the length of each video sequence was between 50 and 150 frames. The feature information of the target object was extracted according to the prior art and the solution provided in the embodiment of the present application, and the target tracking effect was as follows:

[0164] When feature extraction is performed according to the prior art, if the template size is 64, the search area is scaled while keeping the aspect ratio unchanged to generate a template image, then the shorter side of the template image is 64, and the feature information extracted from the template image consists of 31-dimensional FHOG features and 15-dimensional Lab features.

[0165] When feature extraction is performed according to the scheme of the embodiment of the present application, if the template size is 64x64, the search area can be scaled to generate a 64x64 template image, and the feature information extracted from the template image consists of 22-dimensional FHOG features and 5-dimensional Lab features.

[0166] When the experiment is performed according to the above example, the experimental test results shown in Table 3 below can be obtained.

[0167] Table 3

[0168] Accuracy Failure rate Time complexity (ms) Prior art 91.97％ 11.63％ 2.85 Embodiments of the present application 91.18％ 10.68％ 1.82

[0169] As can be seen from Table 3, compared with the prior art solution, the solution provided by the embodiment of the present application reduces the time complexity of the tracking algorithm by 36.14%, reduces the accuracy of the tracking algorithm by 0.79%, and increases the failure rate of the tracking algorithm by 0.95%. Although the tracking effect of the algorithm has been reduced, the degree of reduction is within an acceptable range, and compared with the degree of reduction in time complexity, the overall effect of the embodiment of the present application is very good, and both the tracking effect and time complexity are well taken into account.

[0170] In addition, in order to more intuitively reflect the optimization effect of the embodiment of the present application on time complexity, please refer to Figure 4 The line graph shown shows the comparison of the tracking time consumption of each video sequence in 206 video sequences. Among them, 1 represents the solution provided by the prior art, and 2 represents the solution provided by the embodiment of the present application.

[0171] It is understandable that after the client obtains the video sequence, it can directly track the target object in the video sequence according to the solution for obtaining the feature information of the target object provided in the embodiment of the present application. Alternatively, as a preferred solution, the client can use the solution provided in the embodiment of the present application to track the target object in the video sequence when it determines that the computing power consumed by the feature information extraction process needs to be reduced according to actual application requirements.

[0172] In one way, the available processing resources of the terminal device associated with the client can be obtained, for example, the available resources of the CPU of the terminal device, and whether it is necessary to save the computing power of the device at present can be determined accordingly. Specifically, the first threshold value can be determined according to the use needs. If the available processing resources of the terminal device are less than the first threshold value, it can be determined that the processing resources of the terminal device are tight, and it is necessary to reduce the computing power consumed by the feature information extraction process through the solution provided in the embodiment of the present application, so as to alleviate the consumption of device processing resources by target tracking. If it is determined according to the first threshold that the processing resources of the device are relatively idle, other solutions (for example, the solution provided by the prior art for obtaining the feature information of the target object by using a template that only sets the short side size information) can be used to extract the feature information of the target object.

[0173] Alternatively, in another way, the number information of the target objects to be tracked and the second threshold value determined according to the use requirements can be obtained. If the number of target objects is greater than the second threshold value, it can be determined that the computing power consumed by the feature information extraction process of each target object needs to be reduced. Taking the video sequence as a road traffic video as an example, for the more congested section A, when it is determined that the computing power of the device needs to be saved, the feature information extraction scheme provided by the embodiment of the present application can be used to achieve target tracking; for the section B with fewer vehicles, other schemes can be used to extract the feature information of the target object to achieve target tracking. Alternatively, the client can also determine whether to enable the scheme provided by the embodiment of the present application according to the busyness of the road traffic at different time periods, so as to save the computing power of the device. For example, for section C, the traffic volume is large during the rush hour, and the feature information extraction scheme provided by the embodiment of the present application can be used to achieve target tracking; in the idle period with relatively small traffic volume, other schemes can be used to extract the feature information of the target object to achieve target tracking.

[0174] It can be understood that the solution provided in the embodiment of the present application can extract the feature information of the target object in the video sequence shot in different scenes to achieve target tracking. Figure 5 The flowchart shown explains the target tracking process in this scenario.

[0175] Example 2

[0176] S201: The client obtains size information of at least one available template, where the size information includes width information of the available template and height information of the available template.

[0177] The implementation process of S201 can refer to the introduction of S101 above, and will not be described in detail here.

[0178] S202: If a video sequence related to a vehicle traveling on a road is obtained, a target object to be tracked is determined from the video sequence.

[0179] After the client obtains a video sequence related to vehicles traveling on the road (ie, a road traffic video), it can determine a current frame image from the video sequence, and further determine a target object to be tracked from the current frame image, and perform tracking processing on the target object.

[0180] In one way, the target object may be a vehicle in the current frame image. By tracking the vehicle, it can be determined whether the vehicle has violated the rules during driving. For example, by tracking the vehicle, it can be determined that the vehicle has made a U-turn at an intersection where U-turns are prohibited, and thus it can be determined that the vehicle has violated the rules.

[0181] In another way, the target object can be obstacles around the vehicle, such as other vehicles, pedestrians, road construction facilities, etc. Tracking obstacles around the vehicle helps improve driving safety. For example, based on the obstacle tracking results, information related to the obstacle can be pushed to the driver to remind the driver to avoid the obstacle when necessary; for autonomous vehicles, the vehicle's driving route can be adjusted based on the obstacle tracking results to automatically avoid the obstacle.

[0182] It can be understood that when determining the target object, all objects included in the current frame image can be determined as target objects for tracking processing. Alternatively, the client can determine a corresponding tracking mode for each object included in the current frame image based on different tracking purposes, and determine the target object from the objects included in the current frame image according to the tracking mode information for tracking processing. As an example, the tracking mode may include: objects that need to be tracked and objects that do not need to be tracked.

[0183] Taking the autonomous driving scenario as an example, in order to obtain the distribution of obstacles around the autonomous driving vehicle, the client can determine all objects included in the current frame image as target objects that need to be tracked, and use the solution provided in the embodiment of the present application to extract the feature information of the target objects for target tracking.

[0184] Alternatively, in order to perform obstacle avoidance, the client may determine objects that are close to the autonomous driving vehicle and may affect the normal driving of the autonomous driving vehicle as target objects that need to be tracked; or, based on the relative moving direction between the autonomous driving vehicle and the obstacle, objects that are gradually approaching the autonomous driving vehicle may be determined as target objects that need to be tracked, and the solution provided in the embodiments of the present application may be used to extract feature information of the target object for target tracking.

[0185] As can be seen from the above description, both of the above methods use the objects included in the current frame image as the basic granularity to determine the target object. As an example, the granularity can be further subdivided to determine a specific area of ​​the vehicle as the target object to be tracked.

[0186] The specific area of ​​the vehicle may be a passenger area. For example, the passenger area may include the location of the driver and / or the location of the passenger. The passenger area may be determined in the current frame image based on the relative position relationship between the vehicle and the passenger area, and then the passenger area may be used as the target object for tracking processing to determine whether there are any violations such as making or receiving phone calls (mainly referring to the driver), not wearing a seat belt, and overloading of passengers during the driving of the vehicle.

[0187] Alternatively, the specific area of ​​the vehicle may be a cargo area. Similarly, the cargo area may be determined in the current frame image based on the relative positional relationship between the vehicle and the cargo area, and then the cargo area may be tracked as the target object to determine whether there are any violations such as over-height or over-width during the vehicle's driving.

[0188] It should be noted that, in this embodiment, the video sequence related to the vehicles traveling on the road can be collected by the on-board camera installed on the vehicle and then submitted to the client. Correspondingly, the client can track and process the obstacles around the vehicle accordingly to ensure the safety of the vehicle. Alternatively, the video sequence can be collected by the roadside equipment deployed on the road and then submitted to the client. For example, the perception base station in the intelligent vehicle-road cooperative system, the camera used to shoot violations in the road monitoring system, etc., can all collect video sequences related to vehicles traveling on the road and submit them to the client. Similarly, the client can track and process the obstacles around the vehicle accordingly to ensure the safety of the vehicle. In addition, after the client obtains legal authorization, it can also track the vehicle and specific areas of the vehicle to determine whether there are any violations during the vehicle's driving.

[0189] Preferably, when it is determined that a vehicle has violated a traffic rule, the client can submit the vehicle's identification information and violation information to the server, so that the server can implement safe and intelligent traffic management accordingly. For example, after receiving the information submitted by the client, the server can prompt relevant personnel to check the violation and issue a ticket if necessary.

[0190] S203: Determine a target template from the at least one available template, obtain feature information of the target object using the target template, and perform tracking processing on the target object.

[0191] The implementation process of S203 can refer to the introduction of S102 to S105 above, which will not be described in detail here.

[0192] In addition, the present invention also provides a device for obtaining feature information of a target object, see Figure 6 The device is applied to a client and includes:

[0193] A size information obtaining unit 301 is used to obtain size information of at least one available template, wherein the size information includes width information of the available template and height information of the available template;

[0194] A search area obtaining unit 302 is used to obtain a search area corresponding to the target object to be tracked in the current frame image;

[0195] An image information storage unit 303 is used to create a corresponding storage area for the search area and save the image information associated with the search area to the storage area;

[0196] A target template determining unit 304, configured to determine a target template from the available templates;

[0197] A template image generating unit 305, configured to generate a template image by performing scaling processing on the image information in the storage area according to the size information of the target template;

[0198] The feature information extraction unit 306 is used to extract feature information related to the target object from the template image, so as to track the target object in the next frame image according to the feature information.

[0199] If it is necessary to reduce the computing power consumed by extracting the feature information, the feature information of the target object is obtained by using the target template determined from the available templates.

[0200] If the available processing resources of the terminal device associated with the client are less than a first threshold, it is determined that the computing power consumed by feature information extraction needs to be reduced.

[0201] According to the fact that the number of target objects to be tracked is greater than the second threshold, it is determined that the computing power consumed by feature information extraction needs to be reduced.

[0202] Wherein, the feature information extraction unit is specifically used for:

[0203] If the feature information is FHOG feature information, the code line for extracting unsigned HOG feature information is deleted in the implementation code for extracting the FHOG feature, so as to reduce the number of feature dimensions included in the FHOG feature information.

[0204] Wherein, the feature information extraction unit is specifically used for:

[0205] If the feature information is Lab feature information, clustering is performed on the original centroids to obtain new centroids, and the number of the new centroids is less than the number of the original centroids;

[0206] The Lab feature information is extracted from the template image according to the new centroid to reduce the number of feature dimensions included in the Lab feature information.

[0207] Wherein, the device further comprises:

[0208] A first down-sampling processing unit, configured to perform down-sampling processing on the current frame image to generate a processed current frame image when the search area exceeds the boundary of the current frame image;

[0209] A boundary filling unit, used for filling the boundary of a new search area in the current frame image after the processing;

[0210] The image information storage unit is specifically used to: store the image information associated with the filled new search area into the storage area.

[0211] Wherein, the device further comprises:

[0212] A filling information obtaining unit, configured to obtain row number information and column number information for filling the border of the search area when it is determined that the search area exceeds the border of the current frame image;

[0213] The first down-sampling processing unit is specifically configured to: if the row number information and the column number information meet a first preset condition, perform down-sampling processing on the current frame image.

[0214] The first preset condition is that the maximum value of the row number information and the column number information exceeds a first threshold, and the first threshold is determined by the minimum value of the width information and the height information of the current frame image, and a first coefficient;

[0215] Then, according to the maximum value of the row number information and the column number information, and the first threshold, a downsampling multiple for downsampling the current frame image is determined.

[0216] The first preset condition is that the minimum value of the row number information and the column number information exceeds a second threshold value, and the second threshold value is determined by the minimum value of the width information and the height information of the current frame image and a second coefficient;

[0217] Then, according to the maximum value of the row number information and the column number information, and the second threshold, a downsampling multiple for downsampling the current frame image is determined.

[0218] Wherein, the device further comprises:

[0219] A second downsampling processing unit, configured to perform downsampling processing on the current frame image to generate a processed current frame image when the size information of the search area meets a second preset condition;

[0220] The image information storage unit is specifically used for:

[0221] The image information associated with the new search area in the processed current frame image is saved in the storage area.

[0222] The second preset condition is that the maximum value of the height information and the width information of the search area exceeds a third threshold, and the third threshold is determined by the minimum value of the width information and the height information of the current frame image and a third coefficient;

[0223] The downsampling multiple for downsampling the current frame image is determined according to the third coefficient.

[0224] The second preset condition is that the minimum value of the distance between the boundary of the search area and the corresponding boundary of the current frame image is less than a fourth threshold, and the fourth threshold is determined by the size information of the current frame image;

[0225] The preset side length of the new search area after downsampling is obtained, and the downsampling multiple for downsampling the current frame image is determined based on the minimum value of the width information and the height information of the search area and the preset side length, and the preset side length is not greater than the length of the shortest side of the target template.

[0226] Wherein, the device further comprises:

[0227] The fourth threshold value obtaining unit is specifically used for:

[0228] Obtaining a ratio between the height information of the current frame image and the height information of the search area, and a ratio between the width information of the current frame image and the width information of the search area;

[0229] Determine the side length of the current frame image corresponding to the maximum ratio as the basic side length;

[0230] The fourth threshold is obtained according to the length information of the basic side length and a fourth coefficient.

[0231] The shape of the available template is a square, and the length of the side of the available template ranges from 64 to 128 pixels.

[0232] The shape of the available template is a rectangle, and the length of the short side of the available template is the minimum value to ensure the effectiveness of the cosine window function, and the cosine window function is used to smooth the extracted feature information.

[0233] The size information obtaining unit is specifically used for:

[0234] If the size information of the available templates is stored in the server, the server is accessed to obtain the size information of the at least one available template.

[0235] The size information obtaining unit is specifically used for:

[0236] If the size information of the available templates is pre-packaged in the implementation code of the client, the implementation code is executed to obtain the size information of the at least one available template.

[0237] Wherein, the device further comprises:

[0238] The newly added size information obtaining unit is used to provide an operation option for obtaining the size information of the newly added available template, and save the size information of the newly added available template after obtaining it through the operation option.

[0239] Wherein, if the current frame image is an initial frame image, the target template determination unit is specifically used to:

[0240] Obtaining a target aspect ratio of a search area in the initial frame image;

[0241] The aspect ratio of each available template is obtained, and the available template with a height-to-width ratio close to the target height-to-width ratio is determined as the target template.

[0242] In addition, the present application also provides a target tracking processing device, see Figure 7 The device is applied to a client and includes:

[0243] A size information obtaining unit 401 is used to obtain size information of at least one available template, wherein the size information includes width information of the available template and height information of the available template;

[0244] A target object determination unit 402 is used to determine a target object to be tracked from a video sequence when a video sequence related to a vehicle traveling on a road is obtained;

[0245] The tracking processing unit 403 is used to determine a target template from the available templates, obtain feature information of the target object using the target template, and perform tracking processing on the target object.

[0246] Wherein, if at least two objects are determined from the video sequence,

[0247] The organizations also include:

[0248] A tracking mode determination unit, used to determine the tracking mode information corresponding to each object;

[0249] The target object determination unit is specifically configured to determine the object whose feature information needs to be extracted, as the target object to be tracked, according to the tracking mode information.

[0250] In addition, an embodiment of the present application further provides an electronic device, including:

[0251] one or more processors; and

[0252] A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the following operations are performed:

[0253] Obtaining size information of at least one available template, wherein the size information includes width information of the available template and height information of the available template;

[0254] Obtaining a search area corresponding to the target object to be tracked in the current frame image, creating a corresponding storage area for the search area, and saving image information associated with the search area to the storage area;

[0255] Determine a target template, and perform scaling processing on the image information in the storage area according to the size information of the target template to generate a template image;

[0256] Feature information related to the target object is extracted from the template image so as to track the target object in the next frame image according to the feature information.

[0257] And an electronic device, comprising:

[0258] one or more processors; and

[0259] A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the following operations are performed:

[0260] Obtaining size information of at least one available template, wherein the size information includes width information of the available template and height information of the available template;

[0261] If a video sequence related to a vehicle traveling on a road is obtained, a target object to be tracked is determined from the video sequence;

[0262] A target template is determined from the at least one available template, feature information of the target object is obtained using the target template, and tracking processing is performed on the target object.

[0263] in, Figure 8 The architecture of an electronic device is exemplarily shown. For example, device 500 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, an aircraft, etc.

[0264] Reference Figure 8 , device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 503 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0265] The processing component 502 generally controls the overall operation of the device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the method provided by the technical solution of the present disclosure. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 503 and the processing component 502.

[0266] The memory 504 is configured to store various types of data to support operations on the device 500. Examples of such data include instructions for any application or method operating on the device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0267] Power supply component 506 provides power to various components of device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0268] The multimedia component 503 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 503 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0269] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the device 500 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0270] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0271] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the device 500. For example, the sensor assembly 514 can detect the open / closed state of the device 500, the relative positioning of components, such as the display and keypad of the device 500, and the sensor assembly 514 can also detect the position change of the device 500 or a component of the device 500, the presence or absence of user contact with the device 500, the orientation or acceleration / deceleration of the device 500, and the temperature change of the device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0272] The communication component 516 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0273] In an exemplary embodiment, the device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0274] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the above instructions can be executed by a processor 520 of the device 500 to complete the method provided by the technical solution of the present disclosure. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0275] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0276] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0277] The above is a detailed introduction to the method, device and electronic device for obtaining the characteristic information of the target object provided by the present application. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for obtaining characteristic information of a target object, characterized in that: include: The client obtains size information of at least one available template, where the size information is used to limit the size of the generated template image, and the size information includes width information of the available template and height information of the available template; Obtaining a search area corresponding to the target object to be tracked in the current frame image, creating a corresponding storage area for the search area, and saving image information associated with the search area to the storage area; Determine a target template from the available templates, and perform scaling processing on the image information in the storage area according to the size information of the target template to generate a template image so that the size of the template image is the size of the target template; Feature information related to the target object is extracted from the template image so as to track the target object in the next frame image according to the feature information.

2. The method according to claim 1, characterized in that If it is necessary to reduce the computing power consumed by extracting the feature information, the feature information of the target object is obtained by using the target template determined from the available templates.

3. The method according to claim 2, characterized in that If the available processing resources of the terminal device associated with the client are less than the first threshold, it is determined that the computing power consumed by feature information extraction needs to be reduced.

4. The method according to claim 2, characterized in that: According to the fact that the number of target objects to be tracked is greater than the second threshold, it is determined that the computing power consumed by feature information extraction needs to be reduced.

5. The method according to claim 1, characterized in that The step of extracting feature information related to the target object from the template image includes: If the feature information is FHOG feature information, the code line for extracting unsigned HOG feature information is deleted in the implementation code for extracting the FHOG feature, so as to reduce the number of feature dimensions included in the FHOG feature information.

6. The method according to claim 1, characterized in that The step of extracting feature information related to the target object from the template image includes: If the feature information is Lab feature information, clustering is performed on the original centroids to obtain new centroids, and the number of the new centroids is less than the number of the original centroids; The Lab feature information is extracted from the template image according to the new centroid to reduce the number of feature dimensions included in the Lab feature information.

7. The method according to claim 1, characterized in that Also includes: If the search area exceeds the boundary of the current frame image, down-sampling the current frame image to generate a processed current frame image; Filling the boundaries of a new search area in the current frame image after the processing; Saving the image information associated with the search area to the storage area includes: saving the image information associated with the new search area after filling to the storage area.

8. The method according to claim 7, characterized in that After determining that the search area exceeds the boundary of the current frame image, the method further includes: Obtaining information on the number of rows and columns for filling the border of the search area; The down-sampling the current frame image includes: if the row number information and the column number information meet a first preset condition, down-sampling the current frame image.

9. The method according to claim 8, characterized in that The first preset condition is that the maximum value of the row number information and the column number information exceeds a first threshold, and the first threshold is determined by the minimum value of the width information and the height information of the current frame image, and a first coefficient; Then, according to the maximum value of the row number information and the column number information, and the first threshold, a downsampling multiple for downsampling the current frame image is determined.

10. The method according to claim 8, characterized in that The first preset condition is that the minimum value of the row number information and the column number information exceeds a second threshold, and the second threshold is determined by the minimum value of the width information and the height information of the current frame image and a second coefficient; Then, according to the maximum value of the row number information and the column number information, and the second threshold, a downsampling multiple for downsampling the current frame image is determined.

11. The method according to claim 1, characterized in that: Also includes: If the size information of the search area meets the second preset condition, down-sampling the current frame image to generate a processed current frame image; Saving the image information associated with the search area to the storage area includes: The image information associated with the new search area in the processed current frame image is saved in the storage area.

12. The method according to claim 11, characterized in that The second preset condition is that the maximum value of the height information and the width information of the search area exceeds a third threshold, and the third threshold is determined by the minimum value of the width information and the height information of the current frame image and a third coefficient; The downsampling multiple for downsampling the current frame image is determined according to the third coefficient.

13. The method according to claim 11, characterized in that The second preset condition is that the minimum value of the distance between the boundary of the search area and the corresponding boundary of the current frame image is less than a fourth threshold, and the fourth threshold is determined by the size information of the current frame image; The preset side length of the new search area after downsampling is obtained, and the downsampling multiple for downsampling the current frame image is determined based on the minimum value of the width information and the height information of the search area and the preset side length, and the preset side length is not greater than the length of the shortest side of the target template.

14. The method according to claim 13, characterized in that The fourth threshold is obtained according to the size information of the current frame image in the following manner: Obtaining a ratio between the height information of the current frame image and the height information of the search area, and a ratio between the width information of the current frame image and the width information of the search area; Determine the side length of the current frame image corresponding to the maximum ratio as the basic side length; The fourth threshold is obtained according to the length information of the basic side length and a fourth coefficient.

15. The method according to claim 1, characterized in that The shape of the available template is a square, and the length of the side of the available template ranges from 64 to 128 pixels.

16. The method according to claim 1, characterized in that The shape of the available template is a rectangle, and the length of the short side of the available template is the minimum value to ensure the effectiveness of the cosine window function, and the cosine window function is used to smooth the extracted feature information.

17. The method according to claim 1, characterized in that The obtaining of size information of at least one available template comprises: If the size information of the available templates is stored in the server, the server is accessed to obtain the size information of the at least one available template.

18. The method according to claim 1, characterized in that The obtaining of size information of at least one available template comprises: If the size information of the available templates is pre-packaged in the implementation code of the client, the implementation code is executed to obtain the size information of the at least one available template.

19. The method according to claim 17 or 18, characterized in that Also includes: An operation option for obtaining size information of a newly added available template is provided, and the size information of the newly added available template is saved after being obtained through the operation option.

20. The method according to claim 1, characterized in that If the current frame image is an initial frame image, determining the target template includes: Obtaining a target aspect ratio of a search area in the initial frame image; The aspect ratio of each available template is obtained, and the available template with a height-to-width ratio close to the target height-to-width ratio is determined as the target template.

21. A target tracking processing method, characterized in that: include: The client obtains size information of at least one available template, where the size information is used to limit the size of the generated template image, and the size information includes width information of the available template and height information of the available template; If a video sequence related to a vehicle traveling on a road is obtained, a target object to be tracked is determined from the video sequence; Determine a target template from the available templates, obtain feature information of the target object using the target template, and perform tracking processing on the target object; Among them, the use of the target template to obtain the characteristic information of the target object includes: obtaining the search area corresponding to the target object in the video sequence, and saving the image information associated with the search area to a storage area created for the search area, scaling the image information in the storage area according to the size information of the target template to generate a template image, so that the size of the template image is the size of the target template, and then extracting the characteristic information related to the target object from the template image.

22. The method according to claim 21, characterized in that The target objects are obstacles around the vehicle.

23. The method according to claim 21, characterized in that The target object is a specific area of ​​the vehicle.

24. The method according to claim 21, characterized in that If at least two objects are determined from the video sequence, The method further comprises: Determine tracking mode information corresponding to each object; The object for which feature information extraction is required, indicated by the tracking mode information, is determined as the target object to be tracked.

25. A device for obtaining characteristic information of a target object, characterized in that: Applied to the client, including: A size information obtaining unit, used to obtain size information of at least one available template, wherein the size information is used to limit the size of the generated template image, and the size information includes width information of the available template and height information of the available template; A search area obtaining unit, used to obtain a search area corresponding to the target object to be tracked in the current frame image; An image information storage unit, used to create a corresponding storage area for the search area, and save the image information associated with the search area to the storage area; A target template determining unit, configured to determine a target template from the available templates; A template image generating unit, configured to generate a template image by scaling the image information in the storage area according to the size information of the target template, so that the size of the template image is the size of the target template; The feature information extraction unit is used to extract feature information related to the target object from the template image, so as to track the target object in the next frame image according to the feature information.

26. A target tracking processing device, characterized in that: Applied to the client, including: A size information obtaining unit, used to obtain size information of at least one available template, wherein the size information is used to limit the size of the generated template image, and the size information includes width information of the available template and height information of the available template; a target object determination unit, configured to determine a target object to be tracked from a video sequence when a video sequence related to a vehicle traveling on a road is obtained; A tracking processing unit is used to determine a target template from the available templates, obtain feature information of the target object using the target template, and perform tracking processing on the target object, specifically comprising: obtaining a search area corresponding to the target object in the video sequence, and saving image information associated with the search area to a storage area created for the search area, scaling the image information in the storage area according to the size information of the target template to generate a template image so that the size of the template image is the size of the target template, and then extracting feature information related to the target object from the template image.

27. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the following operations are performed: Obtaining size information of at least one available template, where the size information is used to limit the size of the generated template image, and the size information includes width information of the available template and height information of the available template; Obtaining a search area corresponding to the target object to be tracked in the current frame image, creating a corresponding storage area for the search area, and saving image information associated with the search area to the storage area; Determine a target template, and perform scaling processing on the image information in the storage area according to the size information of the target template to generate a template image so that the size of the template image is the size of the target template; Feature information related to the target object is extracted from the template image so as to track the target object in the next frame image according to the feature information.

28. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the following operations are performed: Obtaining size information of at least one available template, where the size information is used to limit the size of the generated template image, and the size information includes width information of the available template and height information of the available template; If a video sequence related to a vehicle traveling on a road is obtained, a target object to be tracked is determined from the video sequence; Determine a target template from the available templates, obtain feature information of the target object using the target template, and perform tracking processing on the target object; Among them, the use of the target template to obtain the characteristic information of the target object includes: obtaining the search area corresponding to the target object in the video sequence, and saving the image information associated with the search area to a storage area created for the search area, scaling the image information in the storage area according to the size information of the target template to generate a template image, so that the size of the template image is the size of the target template, and then extracting the characteristic information related to the target object from the template image.

Images