Target counting method and device, electronic equipment and storage medium

Abstract

The application provides a target counting method and device, electronic equipment and storage medium, wherein the method comprises: determining a monitoring image sequence, performing target identification on each frame image in the monitoring image sequence to obtain a target position, a target feature and / or a target category of a detected target in each frame image; and performing target counting on the monitoring image sequence based on a position correlation degree between the target positions of each detected target in consecutive multiple frame images in the monitoring image sequence, and a feature correlation degree between the target features of each detected target and / or a category correlation degree between the target categories of each detected target. The method, device, electronic equipment and storage medium provided by the application can construct the space-time correlation between each detected target through the correlation degrees of the positions, features and categories between each detected target in consecutive multiple frame images, improve the accuracy of judging whether each detected target is actually the same detected target, and further improve the identification effect of the detected target.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a target counting method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the knitting industry, the production process of grey fabric is divided into weft knitting and warp knitting. During the weft knitting process, defects may occur on the fabric surface due to damage to the knitting needles, resulting in substandard fabric.

[0003] Currently, there are two main methods for detecting substandard fabric in weft knitting: Method 1) Using infrared lasers, the knitting needles are judged based on whether light is reflected, thus determining the production status; Method 2) Using imaging equipment, the captured image is segmented using traditional algorithms. The rotation speed and initial fabric information are recorded during the registration phase. The segmented target is compared with the initial fabric information and the defect target location is calculated.

[0004] However, the infrared laser method in method 1) has poor stability. Due to the large amount of floating cotton fibers in the weft knitting production environment, it is easy to block the receiver and cause misidentification, resulting in frequent shutdowns and affecting production. In method 2), the traditional algorithm for segmenting targets does not establish the spatiotemporal relationship of targets in each frame, making it difficult to determine whether the targets in each frame are the same, which in turn leads to a great limitation in the target recognition effect. Summary of the Invention

[0005] This invention provides a target counting method, apparatus, electronic device, and storage medium to address the shortcomings of traditional algorithms in target recognition in terms of applicability, ease of use, and poor recognition effect.

[0006] This invention provides a target counting method, comprising:

[0007] A monitoring image sequence is determined, and target recognition is performed on each frame of the monitoring image sequence to obtain the target location, target features and / or target category of the detected target in each frame of the image;

[0008] Based on the positional correlation between the target locations of each detected target in multiple consecutive frames of the monitored image sequence, the feature correlation between the target features of each detected target, and / or the category correlation between the target categories of each detected target, the monitored image sequence is counted.

[0009] According to a target counting method provided by the present invention, the target counting of the monitoring image sequence based on the positional correlation between the target positions of each detected target in a series of consecutive frames in the monitoring image sequence, the feature correlation between the target features of each detected target, and / or the feature category degree between the target categories of each detected target, includes:

[0010] Based on the positional correlation between the target positions of each detected target in multiple consecutive frames of the monitored image sequence, the feature correlation between the target features of each detected target and / or the category correlation between the target categories of each detected target, the target features of each detected target in the corresponding image are fused to obtain the fused features of each detected target;

[0011] Based on the fusion features of each detected target, target counting is performed on the monitoring image sequence.

[0012] According to a target counting method provided by the present invention, the step of counting targets in the monitoring image sequence based on the fusion features of each detected target includes:

[0013] The fusion feature of any one of the detected targets is concatenated with the encoded feature of the target position of any one of the detected targets to obtain the concatenated feature of any one of the detected targets;

[0014] Based on the stitching features of each detected target, target counting is performed on the monitoring image sequence.

[0015] According to a target counting method provided by the present invention, the category correlation between the target categories of each detected target is determined based on whether the target categories of each detected target are consistent.

[0016] According to a target counting method provided by the present invention, the positional correlation between the target positions of each detected target is determined based on the correlation between the center coordinates of the target positions of each detected target and the correlation between the dimensions of the target positions.

[0017] According to a target counting method provided by the present invention, the step of determining a monitoring image sequence, performing target recognition on each frame of the monitoring image sequence, and obtaining the target location, target features, and / or target category of the detected target in each frame of the image includes:

[0018] Feature extraction is performed on each frame of the image to obtain the image features of each frame of the image;

[0019] Target recognition is performed on the image features of each frame to obtain the target location and target category of the detected target in each frame;

[0020] Based on the target location of the detected target in each frame image and the upsampling features of each frame image, the target features of the detected target in each frame image are determined; the upsampling features of each frame image are obtained by upsampling the image features of each frame image.

[0021] According to a target counting method provided by the present invention, the monitoring image sequence is obtained by taking pictures of the object to be monitored from multiple angles within one rotation cycle.

[0022] The present invention also provides a target counting device, comprising:

[0023] The determination module is used to determine the monitoring image sequence, perform target recognition on each frame of the monitoring image sequence, and obtain the target location, target features and / or target category of the detected target in each frame of the image;

[0024] The counting module is used to count targets in the monitoring image sequence based on the positional correlation between the target positions of each detected target in multiple consecutive frames of images in the monitoring image sequence, the feature correlation between the target features of each detected target, and / or the category correlation between the target categories of each detected target.

[0025] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the target counting methods described above.

[0026] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the target counting methods described above.

[0027] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the target counting methods described above.

[0028] The target counting method, apparatus, electronic device, and storage medium provided by the present invention construct the temporal and spatial correlation between each detected target by analyzing the position, features, and category of each detected target in multiple consecutive frames of images. Based on this, the target is counted in the monitoring image sequence, which improves the accuracy of determining whether each detected target is the same detected target in reality, thereby improving the reliability and accuracy of counting the detected targets. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is one of the flowcharts illustrating the target counting method provided by the present invention;

[0031] Figure 2 This is the second flowchart of the target counting method provided by the present invention;

[0032] Figure 3 This is the third flowchart of the target counting method provided by the present invention;

[0033] Figure 4 This is a flowchart illustrating the method for obtaining the location, category, and features of a detection target provided by the present invention;

[0034] Figure 5 This is a top view of the rotating object to be monitored provided by the present invention;

[0035] Figure 6 This is a structural diagram of the target counting method provided by the present invention;

[0036] Figure 7 This is a schematic diagram of the target counting device provided by the present invention;

[0037] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] Currently, traditional algorithms for target segmentation require prior assessment of the loom speed and storage of initial fabric state information. If the speed changes, recalibration is necessary. Otherwise, when comparing detected targets in two frames after a speed change, the original loom speed is used for conversion. This can lead to identical detected targets in two frames being identified as two separate targets, or different detected targets in two frames being identified as the same target, resulting in incorrect identification. Furthermore, traditional algorithms do not construct the spatiotemporal relationship between detected targets across multiple frames. They rely solely on the pre-assessed loom speed to determine if targets are the same, making it difficult to determine if targets in each frame are identical. For example, if two frames each contain a different detected target that is not actually the same, but their positions are close or identical, the traditional algorithm will identify them as the same target, leading to an error. Therefore, the target identification performance of traditional algorithms is significantly limited.

[0040] Therefore, how to improve the recognition effect of the detected target is a technical problem that urgently needs to be solved in this field.

[0041] To address the above issues, embodiments of the present invention provide a target counting method. Figure 1 This is one of the flowcharts illustrating the target counting method provided by the present invention. For example... Figure 1 As shown, the method includes:

[0042] Step 110: Determine the monitoring image sequence, perform target recognition on each frame of the monitoring image sequence, and obtain the target location, target features and / or target category of the detected target in each frame of the image;

[0043] Step 120: Count targets in the monitoring image sequence based on the positional correlation between the target locations of each detected target in multiple consecutive frames of images in the monitoring image sequence, the feature correlation between the target features of each detected target, and / or the category correlation between the target categories of each detected target.

[0044] When identifying targets in a sequence of monitored images, knowing the spatiotemporal correlation between the identified targets in each frame allows us to determine whether each target is the same. Furthermore, considering that targets can fall into different categories—for example, broken needles, rotten needles, and starting threads on a loom appear differently in images—these three appearances can be categorized into three classes. Additionally, different targets within the same category will exhibit different target features in the images.

[0045] Therefore, embodiments of the present invention identify the detected targets by constructing the spatiotemporal correlation between the detected targets identified in each frame of the image, and by combining the target features and / or the category of the detected targets.

[0046] Specifically, in step 110, target recognition is performed on each frame of the monitoring image sequence based on the input monitoring image sequence to obtain the target location, target features and / or target category of the detected target in each frame.

[0047] It should be noted that the monitored image sequence is a sequence of consecutive frames of the same size in the same direction. Each frame can contain the entire monitored object from different angles. The monitored object can be the fabric surface on a loom or other periodically rotating objects. When monitoring the fabric surface of the loom, the detected target obtained by target recognition can be a white area caused by a broken needle, a dark area caused by random needles, or a starting mark area, etc. The target position of the detected target in each frame can be represented by the coordinates of the center position of the detection box of the detected target in the image and the length and width of the detection box in the image, where the detection box can be the bounding rectangle of the detected target. The target position, target features, and / or target category of the detected target can be obtained by first extracting features from the image where the detected target is located, and then identifying the image features; alternatively, the target position of the detected target can be obtained through image processing algorithms, such as edge detection or image segmentation algorithms, and then feature extraction can be performed based on the segmented target region. This embodiment of the invention does not limit this. The target features of the detected target can also be obtained by first upsampling the image features of the image where the detected target is located, and then indexing the target position of the detected target in the upsampling features. This embodiment of the invention does not limit this.

[0048] In a monitored image sequence, two different frames may contain two different detected targets appearing at the same or similar coordinate positions in both frames, which can easily lead to misjudgment. Furthermore, if a detected target exists in multiple consecutive frames, it will be present in all of those frames, and its position will be relatively close across them. Therefore, the misjudgment in the above situation must be due to a significant time difference between the two selected frames.

[0049] To address the misjudgment caused by the aforementioned situation, this embodiment of the invention ensures that the detected targets in the monitoring image sequence used for target counting have temporal correlation by limiting the time dimension to consecutive multiple frames of images. Based on this, a positional correlation is constructed between the detected targets by combining the coordinate relationships and / or length-width relationships of their positions in the consecutive multiple frames of images; a feature correlation is constructed by the similarity between the target features of the detected targets; and a category correlation is constructed by the similarity of the types of the detected targets or whether their types are the same. Target counting is then performed on the monitoring image sequence. Here, positional correlation indicates the proximity of the detected targets in the image, or the degree of overlap between the detected targets in the image; feature correlation characterizes the similarity of the detected targets in the image presentation; and category correlation characterizes whether the types of the detected targets are the same or similar.

[0050] It should be noted that the positional correlation, feature correlation and / or category correlation of each detected target can be used to determine the spatial correlation of each detected target. Therefore, under the premise of ensuring the temporal correlation of each detected target, the spatial correlation of each detected target can be used to determine whether each detected target is the same target, thereby realizing the target counting of the monitoring image sequence.

[0051] The determination of spatial relevance among the detected targets can be achieved by grouping them based on category relevance or feature relevance, or by grouping them based on category relevance and then by feature relevance. Then, positional relevance is used to determine whether each detected target in the same group is the same actual detected target. The results of this operation are then used to statistically determine the target count of the monitoring image sequence. Alternatively, the relevance weights between detected targets can be calculated using a self-attention mechanism based on the frame order, positional relevance, and category and / or feature relevance. Based on these relevance weights, the target features of each detected target are fused to obtain fused features. Finally, further identification is performed based on the fused features of each detected target to obtain the target count of the monitoring image sequence. This embodiment of the invention does not impose any limitations on these methods.

[0052] The target counting method provided in this invention constructs the temporal and spatial correlation between each detected target by analyzing the position, features, and category of each detected target in multiple consecutive frames of images. Based on this, the method counts targets in the monitored image sequence, thereby improving the accuracy of determining whether each detected target is the same detected target in reality, and thus improving the reliability and accuracy of counting the detected targets.

[0053] In particular, when applying the above target counting method to the inspection of sub-fabric, abnormal areas on the fabric surface can be used as detection targets. By monitoring the changes in the number of detection targets obtained from the target counting, it can be determined whether there are any new abnormal areas, thereby timely interrupting the abnormal operation of the loom.

[0054] Based on the above embodiments, Figure 2 This is the second flowchart illustrating the target counting method provided by the present invention. For example... Figure 2 As shown, step 120 includes:

[0055] Step 121: Based on the positional correlation between the target positions of each detected target in multiple consecutive frames of images in the monitoring image sequence, the feature correlation between the target features of each detected target and / or the category correlation between the target categories of each detected target, the target features of each detected target are fused to obtain the fused features of each detected target.

[0056] Step 122: Count targets in the monitoring image sequence based on the fusion features of each detected target.

[0057] Specifically, based on the positional correlation between the target locations of each detected target in multiple consecutive frames of the monitored image sequence, the feature correlation between the target features of each detected target, and / or the category correlation between the target categories of each detected target, the correlation weight between each detected target is calculated. Based on the correlation weight between each detected target, the target features of each detected target are fused to obtain the fused features of each detected target. Then, the target count is performed on the monitored image sequence by using the fused features of each detected target for identification and statistics.

[0058] It should be noted that the fusion features of each detection target can be obtained by linearly weighting the target features of other detection targets based on the correlation weight between the detection target and other detection targets, or by linearly weighting the target features of the detection target and other detection targets based on the correlation weight between the detection target and other detection targets.

[0059] Target counting in a monitoring image sequence can be performed directly using the fusion features of each target for identification and statistics. Alternatively, the fusion features can be further enhanced, for example, by further concatenating and fusing the fusion features of each target with the target positions of the detected targets corresponding to the fusion features, and then performing identification and statistics on the concatenated and fused features. This embodiment of the invention does not limit this approach. The identification and statistics can involve extracting features from the fusion features or the further concatenated and fused features, then identifying the extracted features to obtain the identification result of whether each detected target is actually the same detected target, and performing statistics based on the identification result to obtain the number of targets contained in the monitoring image sequence.

[0060] The target counting method provided in this invention fuses the target features of each detected target by means of the positional correlation, feature correlation and category correlation between each detected target, thereby enhancing the features of each detected target that is actually the same detected target, making each detected target that is actually the same detected target closer together, improving the accuracy of subsequent feature recognition, and further improving the recognition effect of the detected target.

[0061] Based on the above embodiments, Figure 3 This is the third flowchart illustrating the target counting method provided by this invention. For example... Figure 3 As shown, step 122 includes:

[0062] Step 122-1: Concatenate the fusion feature of any detection target among all detection targets with the encoded feature of the target position of that detection target to obtain the concatenated feature of that detection target;

[0063] Step 122-2: Count targets in the monitoring image sequence based on the splicing features of each detected target.

[0064] Considering that adding the target location information of each detected target to the fusion features of each detected target can further enhance the influence of the location correlation of each detected target on the target count in the subsequent identification and statistical process.

[0065] Specifically, the target position of each target in each detection target is encoded to obtain the encoded feature of the target position of each target. Then, the encoded feature of the target position of each target is concatenated with its own fusion feature to obtain the concatenated feature of each target. Finally, the target count of the monitoring image sequence is performed by recognizing and statistically analyzing the concatenated features of each target.

[0066] It should be noted that the feature concatenation method for the encoded features of the target location and its own fusion features can be either add or concat, and this embodiment of the invention does not impose any restrictions on this. The identification statistics are performed based on the extracted high-level features to determine whether each detected target is actually the same detected target, and then statistics are calculated based on the identification results to obtain the number of targets contained in the monitored image sequence.

[0067] Based on the above embodiments, the class correlation between the target categories of each monitored target in step 110 is determined based on whether the target categories of each detected target are consistent.

[0068] Specifically, if any two detection targets belong to different categories, their category correlation is 0; otherwise, it is 1. Following this logic, the category correlation between any one detection target and the other detection targets can be obtained.

[0069] Based on the above embodiments, the positional correlation between the target positions of each detected target in a series of consecutive frames in the monitoring image sequence in step 110 is determined based on the correlation between the center coordinates and the size correlation between each detected target in a series of consecutive frames in the monitoring image sequence.

[0070] Specifically, the positional correlation between the target positions of each detected target in a series of consecutive frames of images is determined based on the correlation and size correlation between the center coordinates of any two detected targets in a series of consecutive frames of images.

[0071] It should be noted that the correlation between the center coordinates can be determined by the difference in coordinates of the center points of the two detection targets along the same axis, or by the distance between the center points of the two detection targets. This embodiment of the invention does not impose any restrictions on this. The size correlation can be determined by the width and height of the detection boxes of the two detection targets. The positional correlation between the two detection targets is determined by the correlation between the center coordinates and the size correlation. For example, the correlation between the center coordinates and the size correlation can be multiplied to obtain the positional correlation, or the mean or weighted sum of the correlation between the center coordinates and the size correlation can be obtained as the positional correlation.

[0072] Based on the above embodiments, Figure 4 This is a flowchart illustrating the method for obtaining the location, category, and features of a detection target provided by this invention. For example... Figure 4 As shown, step 110 includes:

[0073] Step 111: Extract features from each frame of the image to obtain the image features of each frame;

[0074] Step 112: Perform target recognition on the image features of each frame to obtain the target location and target category of the detected target in each frame;

[0075] Step 113: Determine the target features of each frame image based on the target location of the detected target in each frame image and the upsampling features of each frame image; the upsampling features of each frame image are obtained by upsampling the image features of each frame image.

[0076] Specifically, feature extraction is performed on each frame image, and target recognition is performed on the image features of each frame image to obtain the target position and target category of each detected target in each frame image. Then, the image features of each frame image are upsampled to obtain the upsampled features of each frame image. Then, based on the target position of each detected target, the upsampled features of the frame image corresponding to each detected target are indexed to obtain the target features of each detected target.

[0077] It should be noted that indexing based on the target location of the detected target in the upsampled features of the frame image can be achieved by corresponding the x-coordinate of a point in the target location region of the detected target to the column number of the upsampled features of the frame image corresponding to the detected target, and the y-coordinate to the row number of the upsampled features of the frame image corresponding to the detected target, and so on. The target features of the detected target can be obtained by indexing all points in the target location region of the detected target, or by indexing each vertex of the search box of the target location of the detected target, and then extracting the features from the upsampled feature region corresponding to the region formed by each vertex. This embodiment of the invention does not limit this approach.

[0078] Based on any of the above embodiments, the monitoring image sequence in step 110 is obtained by taking pictures of the object to be monitored from multiple angles within one rotation cycle.

[0079] It should be noted that the object to be monitored is an object that can rotate horizontally according to its own vertical center line. One rotation cycle is when the object rotates one full circle from the starting mark back to the starting mark. The camera device can continuously capture images of the entire object from various angles as it rotates horizontally. Figure 5 This is a top-view schematic diagram of the rotation of the object to be monitored provided by the present invention. (See diagram below.) Figure 5 As shown, the outer frame represents the surface of the object to be monitored, with a starting mark engraved on it. The arrow indicates the direction of rotation of the object to be monitored. The triangle represents the camera, and the two dashed arrows at the vertices of the triangle represent the maximum angle that the camera can capture.

[0080] Figure 6 This is a structural diagram of the target counting method provided by the present invention. (See diagram below.) Figure 6 As shown, this method uses a loom as an example, but it is not limited to other similar scenarios. In the diagram, the letter N represents a natural number. The method consists of two stages:

[0081] Phase 1:

[0082] Step 611: When the loom rotates once, the fabric surface rotates once, and a monitoring image sequence of one cycle is captured by the camera.

[0083] Step 612: Each frame of the monitored image sequence is input to the encoder, which is responsible for modeling the image space and detecting the features of the target (abnormal target) on the image features. Then, the target location and target category are output by the localization and recognition unit, and the target location and target category are obtained frame by frame. The target location includes the center x-coordinate and y-coordinate, and the width w and height h of the area where the target is located. The target category is identified by c.

[0084] Phase Two:

[0085] Step 621: Based on the target positions of each detected target obtained in step 612, index the target features f of each detected target from the upsampled features of the encoder output image features.

[0086] Step 622: The correlation weights between detected targets in multiple consecutive frames are calculated via a self-attention module to model the spatiotemporal correlation between detected targets in multiple consecutive frames, and the fusion features of each detected target in multiple consecutive frames are output. The correlation weights ω in this self-attention module consist of positional correlation ω1, category correlation ω2, feature correlation ω3, and frame number correlation ω4. Positional correlation measures the positional correlation between two detected targets, category correlation controls matching within the same category, feature correlation measures the similarity of two detected targets in image representation, and frame number correlation ω4 is used to restrict the detected targets for comparison to come from multiple consecutive frames. The specific calculation formula is as follows:

[0087] ω=ω1*ω2*ω3*ω4

[0088] ω1=(1-(x i -x j ) 2 )(1-(y i -y j ) 2 )(1-(w i -w j ) 2 )(1-|h i -h j |)

[0089]

[0090]

[0091]

[0092] In the formula, ω represents the spatiotemporal relationship between the i-th and j-th identified targets, ω1 represents the positional weight of the i-th and j-th identified targets, ω2 represents the type weight of the i-th and j-th identified targets, ω3 represents the similarity between the i-th and j-th identified targets, and ω4 represents the frame weight of the similarity between the i-th and j-th identified targets; where x i It is the x-axis coordinate of the center of the i-th target in the frame image where the i-th target is located. j It is the x-axis coordinate of the center of the j-th identified target in the frame image where the j-th identified target is located; y i It is the y-coordinate of the center of the i-th target in the frame image where the i-th target is located. jw is the y-coordinate of the center of the j-th target in the frame image where the j-th target is located; i w is the width of the center of the i-th target in the frame image where the i-th target is located. j h is the width of the center of the j-th target in the frame image where the j-th target is located; i h is the width of the center of the i-th target in the frame image where the i-th target is located. j c is the width of the center of the j-th identified target in the frame image where the j-th identified target is located; i c is the type of the i-th identified target. j It is the type of the j-th identified target; f i f is the semantic feature of the i-th target to be identified. j Let t be the semantic feature of the j-th target being identified, and T be the transpose matrix; i t is the index of the i-th frame containing the target image in the sequence of images to be detected. j It is the index of the frame image containing the j-th target in the sequence of images to be detected.

[0093] Step 623: The fusion features generated by each detection target are concatenated with the encoded features of the target position corresponding to the fusion features to form a new feature representation for each detection target. The context information of each detection target in the time and space is modeled by a bidirectional RNN, and the high-level semantic feature representation of the entire cycle is output. Finally, the number of detection targets contained in the monitoring image sequence is output by a regression predictor containing multiple fully connected layers.

[0094] It should be noted that the loss function for the iterative training of the first stage of target detection and recognition consists of the mean squared error loss of regression position plus the classification cross-entropy loss. After obtaining the output through forward propagation, the gradient is calculated based on the combined loss function, and feedback training is performed until the model converges. The loss function for the second stage is the mean squared error loss. Similarly, after obtaining the count output through forward propagation, the gradient is calculated, and iterative training is performed until convergence.

[0095] The forward test after training is as follows: input a periodic multi-monitoring image sequence, after encoding and decoding in the target recognition stage, output the possible target location and corresponding target category; after the target spatiotemporal correlation modeling and regression prediction stage, output the target count result of the monitoring image sequence.

[0096] The target counting device provided by the present invention is described below. The target counting device described below can be referred to in correspondence with the target counting method described above.

[0097] Figure 7 This is a schematic diagram of the target counting device provided by the present invention. Figure 7As shown, the target counting device includes: a determination module 710 and a counting module 720.

[0098] in,

[0099] The determination module 710 is used to determine the monitoring image sequence, perform target recognition on each frame of the monitoring image sequence, and obtain the target location, target features and / or target category of the detected target in each frame of the image;

[0100] The counting module 720 is used to count targets in the monitoring image sequence based on the positional correlation between the target positions of each detected target in multiple consecutive frames of images in the monitoring image sequence, the feature correlation between the target features of each detected target, and / or the category correlation between the target categories of each detected target.

[0101] In this embodiment of the invention, the determining module 710 is used to determine the monitoring image sequence, perform target recognition on each frame of the monitoring image sequence, and obtain the target position, target features and / or target category of the detected target in each frame of the image; the counting module 720 is used to count targets in the monitoring image sequence based on the positional correlation between the target positions of each detected target in multiple consecutive frames of the monitoring image sequence, the feature correlation between the target features of each detected target and / or the category correlation between the target categories of each detected target. This realizes the target counting of the monitoring image sequence based on the spatiotemporal correlation, feature correlation and category correlation between each detected target, which improves the accuracy of determining whether each detected target is the same detected target in reality, thereby improving the recognition effect of the detected target.

[0102] Based on any of the above embodiments, the counting module 720 includes:

[0103] The fusion submodule is used to fuse the target features of each detected target based on the positional correlation between the target locations of each detected target in multiple consecutive frames of images in the monitored image sequence, the feature correlation between the target features of each detected target, and / or the category correlation between the target categories of each detected target, to obtain the fused features of each detected target.

[0104] The target counting submodule is used to fuse the features of each detected target and count the targets in the monitored image sequence.

[0105] Based on any of the above embodiments, the target counting submodule includes:

[0106] The splicing submodule is used to splice the fusion feature of any detection target among all detection targets with the encoded feature of the target position of that detection target to obtain the spliced ​​feature of that detection target;

[0107] The image sequence target counting submodule is used to count targets in the monitored image sequence based on the stitching features of each detected target.

[0108] Based on any of the above embodiments, the determining module 710 includes:

[0109] Category Relevance Determination Submodule: This module is used to determine the category relevance between the target categories of each detection target based on whether the target categories of each detection target are consistent.

[0110] Based on any of the above embodiments, the determining module 710 includes:

[0111] The positional correlation determination submodule is used to determine the positional correlation between the target positions of each detected target based on the correlation between the center coordinates of each target position and the correlation between the dimensions of each target position.

[0112] Based on any of the above embodiments, the determining module 710 includes:

[0113] The extraction submodule is used to extract features from each frame of the image to obtain the image features of each frame.

[0114] The location category recognition submodule is used to perform target recognition on the image features of each frame image to obtain the target location and target category of the detected target in each frame image;

[0115] The target feature acquisition submodule is used to determine the target features of the detected target in each frame image based on the target location of the detected target in each frame image and the upsampled features of each frame image; the upsampled features of each frame image are obtained by upsampling the image features of each frame image.

[0116] Based on any of the above embodiments, the determining module 710 includes:

[0117] The image sequence capture submodule is used to capture images of the monitored object from multiple angles within one rotation cycle to obtain a monitoring image sequence.

[0118] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a target counting method, which includes: determining a monitoring image sequence; performing target recognition on each frame of the monitoring image sequence to obtain the target location, target features, and target category of the detected targets in each frame; and counting targets in the monitoring image sequence based on the positional correlation between the target locations of the detected targets in consecutive frames, the feature correlation between the target features of the detected targets, and the category correlation between the target categories of the detected targets.

[0119] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the target counting method provided by the above methods. The method includes: determining a monitoring image sequence; performing target recognition on each frame of the monitoring image sequence to obtain the target position, target features, and target category of the detected target in each frame of the image; and performing target counting on the monitoring image sequence based on the positional correlation between the target positions of each detected target in multiple consecutive frames of the monitoring image sequence, the feature correlation between the target features of each detected target, and the category correlation between the target categories of each detected target.

[0121] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the target counting method provided by the above methods. The method includes: determining a monitoring image sequence; performing target recognition on each frame of the monitoring image sequence to obtain the target position, target features, and target category of the detected targets in each frame of the image; and counting targets in the monitoring image sequence based on the positional correlation between the target positions of the detected targets in multiple consecutive frames of the monitoring image sequence, the feature correlation between the target features of the detected targets, and the category correlation between the target categories of the detected targets.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0123] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target counting method, characterized in that, include: A monitoring image sequence is determined, and target recognition is performed on each frame of the monitoring image sequence to obtain the target location, target features, and target category of the detected target in each frame of the image. The monitoring image sequence is a sequence of consecutive frames of the same size in the same direction. Each frame contains the entire monitored object from different angles. The monitored object is a periodically rotating object. A set of monitoring image sequences is generated every time the monitored object rotates once. Based on the positional correlation between the target locations of each detected target in multiple consecutive frames of the monitored image sequence, the feature correlation between the target features of each detected target, and the category correlation between the target categories of each detected target, the monitored image sequence is counted. Positional relevance is used to represent the proximity of detected targets in an image, or the degree of overlap between detected targets in an image; Feature correlation is used to characterize the similarity of detected targets in image presentation; Type relevance is used to characterize whether the types of detected targets are the same or similar.

2. The target counting method according to claim 1, characterized in that, The method of counting targets in the monitoring image sequence based on the positional correlation between the target locations of each detected target in multiple consecutive frames of the monitoring image sequence, the feature correlation between the target features of each detected target, and the category correlation between the target categories of each detected target includes: Based on the positional correlation between the target positions of each detected target in multiple consecutive frames of the monitored image sequence, the feature correlation between the target features of each detected target, and the category correlation between the target categories of each detected target, the target features of each detected target in the corresponding image are fused to obtain the fused features of each detected target. Based on the fusion features of each detected target, target counting is performed on the monitoring image sequence.

3. The target counting method according to claim 2, characterized in that, The step of counting targets in the monitoring image sequence based on the fusion features of each detected target includes: The fusion feature of any one of the detected targets is concatenated with the encoded feature of the target position of any one of the detected targets to obtain the concatenated feature of any one of the detected targets; Based on the stitching features of each detected target, target counting is performed on the monitoring image sequence.

4. The target counting method according to any one of claims 1 to 3, characterized in that, The class correlation between the target categories of each detection target is determined based on whether the target categories of each detection target are consistent.

5. The target counting method according to any one of claims 1 to 3, characterized in that, The positional correlation between the target positions of each detection target is determined based on the correlation between the center coordinates of the target positions and the correlation between the dimensions of the target positions.

6. The target counting method according to any one of claims 1 to 3, characterized in that, The process of determining the monitoring image sequence, performing target recognition on each frame of the monitoring image sequence, and obtaining the target location, target features, and target category of the detected target in each frame includes: Feature extraction is performed on each frame of the image to obtain the image features of each frame of the image; Target recognition is performed on the image features of each frame image to obtain the target location and target category of the detected target in each frame image; Based on the target location of the detected target in each frame image and the upsampling features of each frame image, the target features of the detected target in each frame image are determined; the upsampling features of each frame image are obtained by upsampling the image features of each frame image.

7. The target counting method according to any one of claims 1 to 3, characterized in that, The monitoring image sequence is obtained by taking pictures of the monitored object from multiple angles within one rotation cycle.

8. A target counting device, characterized in that, include: The determination module is used to determine the monitoring image sequence, perform target recognition on each frame of the monitoring image sequence, and obtain the target location, target features and target category of the detected target in each frame of the image; The monitoring image sequence is a sequence of consecutive frames of the same size in the same direction. Each frame contains the entire monitored object from different angles. The monitored object is a periodically rotating object. A set of monitoring image sequences is generated every time the monitored object rotates once. The counting module is used to count targets in the monitoring image sequence based on the positional correlation between the target positions of each detected target in multiple consecutive frames of images in the monitoring image sequence, the feature correlation between the target features of each detected target, and the category correlation between the target categories of each detected target. Positional relevance is used to represent the proximity of detected targets in an image, or the degree of overlap between detected targets in an image; Feature correlation is used to characterize the similarity of detected targets in image presentation; Type relevance is used to characterize whether the types of detected targets are the same or similar.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the target counting method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the target counting method as described in any one of claims 1 to 7.

Images