Target object statistical methods, target detection methods and neural network training methods

Abstract

The present application relates to a target object statistical method, a target detection method, a neural network training method, a computer device, a computer-readable storage medium and a computer program product, wherein the target object statistical method obtains at least one frame of video image obtained by shooting a target area; performs target detection on the video image to identify at least one target object; tracks the target object to determine the direction of movement of the target object, and generates an identity corresponding to the target object; determines the quantitative statistical information of the target object according to the position distribution between the direction of movement of the target object and a reference line set for the target area, thereby solving the problem that statistical errors are prone to occur in object statistical methods based on computer vision recognition, and improving statistical accuracy.

Description

Technical Field

[0001] The present application relates to the field of animal breeding technology, and in particular to a target object statistical method, a target detection method, a neural network training method, a computer device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In animal transfer inventory and personnel flow detection scenarios, it is necessary to count the number of animals or personnel. Since the target objects are in a state of constant motion, manual counting is difficult and involves more human operations, such as deliberate fabrication of data. Taking animal transfer inventory as an example, the relevant technology provides a pig supervision method based on computer vision recognition. The location and number of pigs are obtained through target detection and target tracking, and the above data is finally uploaded to the enterprise platform, replacing manual inventory with machines to improve fraud problems.

[0003] However, current statistical methods for target objects based on computer vision recognition are prone to statistical errors. Summary of the invention

[0004] Based on this, it is necessary to provide a target object statistical method, target detection method, neural network training method, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a target object counting method, the method comprising:

[0006] Acquire at least one frame of video image obtained by shooting the target area;

[0007] Performing target detection on the video image to identify at least one target object;

[0008] Tracking the target object, determining the moving direction of the target object, and generating an identity corresponding to the target object;

[0009] The quantity statistical information of the target object is determined according to the movement direction of the target object and the position distribution between the reference lines set for the target area.

[0010] In one of the embodiments, performing target detection on the video image to identify at least one target object includes:

[0011] The video image is input into a trained neural network to identify and obtain at least one of the type and position of the target object, wherein the trained neural network is obtained by cropping at least part of the hidden layer of the neural network during training according to a reference ratio, and the reference ratio is determined based on the ratio of the size of the target object in the video image to the size of the entire video image.

[0012] In one embodiment, the quantity statistical information includes a quantity statistical value; and determining the quantity statistical information of the target object according to the movement direction of the target object and the position distribution between the reference lines set for the target area includes:

[0013] Determine whether there is a first target object among the at least one target object, wherein the first target object includes a target object whose motion direction intersects the reference line;

[0014] In response to the presence of a first target object in the at least one target object, the quantity statistical value is determined based on the identity identifier included in the counted target object identifier set.

[0015] In one of the embodiments, a target detection frame is provided for the outer contour of the target object, and the first target object includes a target object whose motion direction intersects the reference line and whose center position of the target detection frame is located on a preset side of the reference line.

[0016] In one embodiment, the target objects include N, where N is a natural number; based on the identity identifiers included in the counted target object identifier set, determining the quantity statistical value includes: performing N counting operations in sequence to obtain the quantity statistical value obtained by the last counting operation, wherein the i+1th counting operation in the N counting operations includes:

[0017] Selecting a first target object on which the counting operation is not performed from the N first target objects;

[0018] In response to the identity identifier of the selected first target object being included in the counted target object identification set, the quantity statistical value obtained by the i-th counting operation is determined as the quantity statistical value obtained by the i+1-th counting operation; or, in response to the identity identifier of the unselected first target object being included in the counted target object identification set, the sum of the quantity statistical value obtained by the i-th counting operation and a preset value is determined as the quantity statistical value obtained by the i+1-th counting operation.

[0019] In one embodiment, the target area includes an entrance and an exit, and in the video image,

[0020] The direction from the entrance to the exit of the target area is not parallel to the reference line, and / or,

[0021] The reference line passes through a preset position of the target area.

[0022] In one of the embodiments, after determining the quantity statistics information of the target object according to the movement direction of the target object and the position distribution between the reference lines set for the target area, the method further includes:

[0023] Determine whether there is an intersection between the movement direction corresponding to the counted target object and the first sub-region, where the first sub-region is a sub-region of the target region;

[0024] When it is determined that the movement direction corresponding to the counted target object has no intersection with the first sub-area, the quantity statistical information is changed.

[0025] In one embodiment, the target object includes animals and people. After acquiring at least one frame of video image obtained by shooting the target area, the method further includes:

[0026] Detecting whether there is an animal in the second sub-area, and detecting whether there is a person in a third sub-area associated with the second sub-area, wherein the second sub-area is a sub-area of ​​the target area, and the third sub-area is a sub-area of ​​the target area;

[0027] When an animal is detected in the second sub-area and a person is detected in the third sub-area, an alarm message is issued.

[0028] In a second aspect, the present application provides a target detection method, the method comprising:

[0029] Acquire at least one frame of video image obtained by shooting the target area;

[0030] The video image is input into a trained neural network to identify and obtain at least one of the type and position of the target object, wherein the trained neural network is obtained by cropping at least part of the hidden layer of the neural network during training according to a reference ratio, and the reference ratio is determined based on the ratio of the size of the target object in the video image to the size of the entire video image.

[0031] In a third aspect, the present application provides a neural network training method, the method comprising:

[0032] Acquire at least one frame of video image obtained by shooting the target area;

[0033] The at least one frame of video image is used as a training sample to train the neural network to be trained, including: cropping at least part of the hidden layer in the neural network during the training process according to a reference ratio, and the reference ratio is determined based on the ratio of the size of the target object in the video image to the size of the entire video image.

[0034] In one embodiment, the neural network includes a first network, the first network includes an input end and an output end, and the first network is configured to extract image features of a target object in the video image; and pruning at least part of the hidden layers in the neural network during training according to a reference ratio includes:

[0035] Dividing the multiple hidden layers in the first network into levels according to the distance between each hidden layer and the input end;

[0036] At least part of the hidden layers divided in the first network is deleted according to the relationship between the ratio and the division level.

[0037] In one embodiment, the first network further comprises a plurality of channels; and pruning at least part of the hidden layers in the neural network during training according to the reference ratio further comprises:

[0038] Obtaining a contribution of the first image feature extracted by each channel to the second image feature output by the output end of the first network;

[0039] Delete channels whose contribution is lower than a first preset threshold.

[0040] In one embodiment, the first network further includes a plurality of residual structure sub-networks, each residual structure sub-network includes at least one channel; and obtaining the contribution of the first image feature extracted by each channel to the second image feature includes:

[0041] The information entropy of each residual structure sub-network is obtained, and the contribution of the first image feature extracted by the channel of each residual structure sub-network to the formation of the second image feature is determined according to the information entropy.

[0042] In one embodiment, pruning at least part of the hidden layers in the neural network during training according to the reference ratio further includes:

[0043] Performing upsampling or downsampling on the multiple branch networks in the first network, wherein the different branch networks after the sampling process have the same number of channels and the same number of hidden layers;

[0044] According to the size of the feature image extracted by each branch network, the multiple branch networks are feature fused to obtain a second network.

[0045] In one embodiment, the neural network further includes a third network, characterized in that the third network is configured to output at least one of the type and position of the target object, wherein an input end of the third network is coupled to an output end of the second network.

[0046] In one embodiment, training the neural network to be trained includes:

[0047] Adjusting the network parameters of the neural network during training, and pruning at least part of the hidden layers of the neural network after the network parameters are adjusted according to a reference ratio to obtain a trained neural network; or,

[0048] At least part of the hidden layers in the neural network during training are pruned according to a reference ratio, and network parameters of the pruned neural network are adjusted to obtain a trained neural network.

[0049] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the first aspect, the second aspect, or the third aspect are implemented.

[0050] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspect, the second aspect, or the third aspect are implemented.

[0051] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in any one of the first aspect, the second aspect, or the third aspect.

[0052] The above-mentioned target object statistical method, target detection method, neural network training method, computer device, computer-readable storage medium and computer program product, wherein the target object statistical method obtains at least one frame of video image obtained by shooting the target area; performs target detection on the video image to identify at least one target object; tracks the target object to determine the movement direction of the target object, and generates an identity corresponding to the target object; determines the quantitative statistical information of the target object based on the movement direction of the target object and the position distribution between the reference line set for the target area, thereby solving the problem that the object statistical method based on computer vision recognition is prone to statistical errors and improving the statistical accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of an application environment of a target object counting method in an embodiment;

[0054] Figure 2 A schematic diagram of a target object counting method in one embodiment;

[0055] Figure 3 Schematic diagram of a target detection method in one embodiment;

[0056] Figure 4 A schematic diagram of setting a reference line in an embodiment Figure 1 ;

[0057] Figure 5 A schematic diagram of setting a reference line in an embodiment Figure 2 ;

[0058] Figure 6 A schematic diagram of a candidate frame in an embodiment;

[0059] Figure 7 Schematic diagram of the relative position relationship between the second sub-area and the third sub-area in one embodiment Figure 1 ;

[0060] Figure 8 Schematic diagram of the relative position relationship between the second sub-area and the third sub-area in one embodiment Figure 2 ;

[0061] Fig. 9 A schematic diagram of a human intrusion alarm in an embodiment;

[0062] Fig.10 A flowchart of a neural network training method in one embodiment;

[0063] Fig.11 A schematic diagram of the structure of a front-end device in an embodiment;

[0064] Fig.12 A schematic diagram of the internal architecture of a front-end device in an embodiment;

[0065] Fig.13 A schematic diagram of an animal transport statistics system in one embodiment;

[0066] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] The object counting method based on computer vision recognition does not further process the results obtained by the tracking algorithm. The statistical range of the target object is relatively vague. When the entrance and exit tracking algorithm misses the tracking, it is easy to make wrong statistics. To solve this problem, this embodiment provides a target object counting method that can be applied to Figure 1 In the application environment shown, the terminal communicates with the server through the network, uploads data to the server, the server processes the data uploaded by the terminal, generates processing results, and stores the data uploaded by the terminal and / or the processing results to the data storage system, and the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers, front-end devices, wherein the front-end device includes a camera or an electronic computing device provided with a camera. The server can be implemented with an independent server or a server cluster composed of multiple servers. The data storage system can be integrated on the server, or it can be placed on the cloud or other network servers. Whether it is a terminal or a server, the target object statistical method provided in this embodiment can be executed.

[0069] In one embodiment, Figure 2 As shown, a method for counting target objects is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the following method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. This embodiment will not be repeated. In this embodiment, the method includes the following steps:

[0070] Step S101, obtaining at least one frame of video image obtained by shooting a target area.

[0071] The target area includes a corridor for transferring the target object from one place to another. The corridor can be a long straight shape or a curved shape. An entrance and an exit are respectively provided at both ends of the corridor so that the target object can enter and exit the corridor. The target area of ​​this embodiment can be an area in an animal transport inventory scene or an area in a personnel flow detection scene.

[0072] Step S102: performing target detection on the video image to identify at least one target object.

[0073] Target detection refers to detecting a target object in a video image based on a deep learning algorithm, and identifying and obtaining at least one of the type and location of the target object. The specific implementation steps of the target detection algorithm can refer to the corresponding embodiments below. Identifying and obtaining the target object means identifying and obtaining at least one of the type and location of the target object. The target object can be an animal or a person, and the animal can be a farmed animal or a wild animal.

[0074] Step S103 , tracking the target object, determining the moving direction of the target object, and generating an identity corresponding to the target object.

[0075] Target tracking refers to extracting image features based on the detection results of target detection, using the extracted image features to estimate the possible position of the target object in the next frame of video image, and tracking the target by determining the position of the target object in the next frame of video image. Among them, a unique identity will be generated for each different target object detected for subsequent identification.

[0076] Step 104 : determining the quantity statistics information of the target objects according to the movement direction of the target objects and the position distribution between the reference lines set for the target area.

[0077] A reference line corresponding to the target area is set in the video image. The reference line can be a line segment set at a certain position in the video image, that is, a reference line is set at the same position of each frame of the video image, which serves as a reference for the activity of the target object. When counting the target objects, it is determined whether there is a target object "passing" the reference line based on the movement direction of the target object and the position distribution between the reference lines. If so, the identity identifier corresponding to the target object is recorded and the target object's quantity statistics information is generated based on the recorded identity identifier. Specifically, each time a target object is detected to "pass" the reference line, the identity identifier of the target object is recorded, and a counting unit is accumulated in the quantity statistics value, wherein each target object is counted at most once to avoid repeated counting when the target object wanders near the reference line. The quantity statistics value can be determined by the following two methods:

[0078] One is that in step S104, each time the target object is detected to "pass" the reference line, it is determined whether the identity of the target object has been counted. If so, the identity is not recorded. If not, the identity is recorded and a counting unit is accumulated in the quantity statistics.

[0079] Another method is to directly record the identity tags corresponding to the target objects that "pass through" the reference line in step S104. The recorded identity tags may overlap. After all target objects are transported (the target objects in the target area disappear or the preset time is reached), the identity tags generated in step S103 and the identity tags recorded in step S104 are intersected, and the identity tags in the intersection are counted, and it is ensured that each identity tag in the intersection is counted only once to obtain a quantitative statistical value. Compared with the former, this method does not need to perform repeated counting judgments every time the target object is detected to "pass through" the reference line, and the execution speed is faster.

[0080] In the above steps S101 to S104, by adding reference line counting logic after target detection and target tracking, the statistical range is made clearer, avoiding the problem of mis-statistics caused by unstable target detection or tracking at the entrance and exit of the target area. At the same time, for the wandering of the target object, an identity corresponding to each target object is set, effectively avoiding repeated statistics. In addition, it can be reused in scenarios where target objects need to be counted, including but not limited to animal transfer inventory and personnel flow detection, with high versatility.

[0081] The related technology is a target object statistical method based on computer vision recognition. The front-end device takes the scene image and transmits the scene image to the back-end device for image processing. During the transmission process, the transmission volume of the scene image is large and the transmission time is long. During the image processing process, the CPU resource overhead is large and the processing time is long, which ultimately leads to poor real-time data processing.

[0082] To solve this problem, in one embodiment, Figure 3 As shown, a target detection method is provided, which can be applied to the above-mentioned target object statistics scenario. This embodiment takes the method applied to the terminal as an example. It can be understood that the following method can also be applied to the server, and can also be applied to the system including the terminal and the server, and is implemented through the interaction between the terminal and the server. This embodiment will not be repeated. In this embodiment, the method includes the following steps:

[0083] Step S201, obtaining at least one frame of video image obtained by shooting the target area;

[0084] Step S202, inputting the video image into the trained neural network to identify and obtain at least one of the type and position of the target object, wherein the trained neural network is obtained by cropping at least part of the hidden layer of the neural network during training according to a reference ratio, and the reference ratio is determined based on the ratio of the size of the target object in the video image to the size of the entire video image.

[0085] In this embodiment, the target area can be an area in an animal transfer inventory scene, or an area in a personnel flow detection scene. The reference ratio refers to the size of the target object in the video image as a percentage of the size of the entire video image. Cropping some hidden layers divided in the neural network according to the reference ratio refers to mapping the reference ratio and the depth relationship of the hidden layer to be cropped. The larger the reference ratio, the deeper the hidden layer to be cropped. For example, animals such as pigs, cattle, sheep, horses, and people are medium to large targets in actual scenes, and do not require too many network layers for target detection. The neural network can be cropped along the depth direction to remove the deep network.

[0086] Among them, taking the object definition of COCO, a general dataset in the field of object detection, as an example, if the image size is 640*480, a small target refers to a target smaller than 32*32 pixels, a medium target refers to a target between 32*32 and 96*96 pixels, and a large target refers to a target larger than 96*96 pixels. Taking animal images as an example, the resolution is about 1920*1080 pixels, which can be converted to a target size definition that is roughly three times that of COCO.

[0087] The depth of the network is determined by dividing the hidden layers in the neural network according to the distance between each hidden layer and the input image. Assuming that the neural network used in this embodiment has 53 layers, it can be considered that the one-third close to the input image is a shallow network, the middle one-third is an intermediate network, and the one-third close to the output image is a deep network.

[0088] In this embodiment, the neural network includes many stacked residual structure sub-networks. Among these structures, the shallow network features have high resolution and retain more complete original image information, which is suitable for extracting small targets. The deep network features have low resolution and are suitable for extracting large targets. The target object of this embodiment is a medium to large target in the actual scene. There is no need for too many network layers for target detection. The neural network can be pruned along the depth direction to remove the deep network. By pruning the neural network, the memory usage and time consumption are greatly reduced while the detection accuracy is not reduced, so that it can run on the chip of the front-end device and realize real-time target detection to meet the real-time requirements of target object statistics, which is convenient for migration and promotion in different scenarios.

[0089] In one embodiment, the neural network can be obtained by modifying the Darknet53 basic network based on the Yolo_v4 algorithm.

[0090] In one embodiment, the ratio of the size of each target object in the video image to the size of the entire video image is within a preset range.

[0091] In one embodiment, the quantity statistical information includes a quantity statistical value; determining the quantity statistical information of the target object according to the position distribution between the movement direction of the target object and the reference line set for the target area includes: determining whether there is a first target object in at least one target object, wherein the first target object includes a target object at which the movement direction and the reference line have an intersection; in response to the presence of the first target object in at least one target object, determining the quantity statistical value based on the identity identifier contained in the counted target object identifier set.

[0092] In this embodiment, when it is detected that there is an intersection between the moving direction of a target object and the reference line, it is determined that the target object "passes" the reference line, and the target object "passing" the reference line is defined as the first target object. The identity of the first target object is obtained, and the target object identity set that has been counted is obtained, and it is checked whether the identity of the first target object exists in the target object identity set. If not, it means that the first target object has not been counted, and the counted quantity statistics value is accumulated by one counting unit, and the identity of the first target object is written into the target object identity set; if so, it means that the first target object has been counted and no processing is performed.

[0093] Further, in one embodiment, the target objects include N, where N is a natural number; based on the identity identifier included in the counted target object identifier set, determining the quantity statistical value includes: executing N counting operations in sequence to obtain the quantity statistical value obtained by the last counting operation, wherein the i+1th counting operation among the N counting operations includes: selecting a first target object on which the counting operation has not been performed from the N first target objects; in response to the identity identifier of the selected first target object included in the counted target object identifier set, determining the quantity statistical value obtained by the i-th counting operation as the quantity statistical value obtained by the i+1th counting operation; or, in response to the identity identifier of the unselected first target object included in the counted target object identifier set, determining the sum of the quantity statistical value obtained by the i-th counting operation and a preset value as the quantity statistical value obtained by the i+1th counting operation.

[0094] In one embodiment, the target area includes an entrance and an exit, and in the video image, the direction from the entrance to the exit of the target area is not parallel to the reference line; or, the reference line passes through a preset position of the target area; or, the direction from the entrance to the exit of the target area is not parallel to the reference line, and the reference line passes through a preset position of the target area. Optionally, the preset position may be the center position of the target area.

[0095] Figure 4 A schematic diagram of setting a reference line is given. In the video image, the reference line spans the aisle and is perpendicular to the extending direction of the aisle.

[0096] Figure 5 Another schematic diagram of setting the reference line is given. In the video image, the reference line spans the aisle and is perpendicular to the extending direction of the aisle. Meanwhile, the reference line also passes through the center position of the aisle.

[0097] Taking the aisle as an example, the reference line passes through the center of the aisle and spans across the aisle. This setting can make the target detected here relatively complete and not easily affected by the unstable detection when the target is incomplete. The center of the aisle can refer to the center position of the entire aisle, such as the position where the distance from the two ends of the aisle is equal, or it can refer to the center position of a part of the aisle, such as the half aisle that appears in the video image.

[0098] During the activity of the target object, the target object may wander back and forth near the reference line, resulting in duplicate recording of the identity mark. It is also possible that the target object returns before completely "passing" the reference line, but the identity mark has been recorded at this time, resulting in errors in the number of target objects.

[0099] To solve this problem, in one embodiment, the identifier corresponding to the target object that "passes through" the reference line in the video image is recorded in the following manner: a target detection frame is set on the outer contour of the target object, and the first target object includes a target object whose motion direction intersects the reference line and the center position of the target detection frame is on a preset side of the reference line.

[0100] In this embodiment, the center position of the candidate box corresponding to each target object in the video image is obtained; it is determined whether the center position of the candidate box is on a preset side of the reference line; when it is determined that the center position of the candidate box is on the preset side of the reference line, the identifier carried by the candidate box is recorded.

[0101] Taking pigs as the target object, Figure 6 A schematic diagram of a candidate box is given. In the video image, the candidate box completely surrounds the target object. The center position of the candidate box refers to the intersection of the vertex lines. The preset side of the reference line refers to the side close to the aisle exit. When the center position of the candidate box is on the preset side of the reference line, it means that at least half of the target object's body "passes" the reference line. At this time, it is more reasonable to record the identity of the target object, avoiding the problem of repeated recording of the identity due to the target object wandering back and forth near the reference line, and also avoiding the problem of the target object returning before completely crossing the reference line, resulting in incorrect counting of the number of target objects.

[0102] When the target area design is more complex, it may not be a long straight aisle, but may be an irregular geometric aisle, resulting in inaccurate counting by relying solely on the target object "passing through" the reference line.

[0103] In order to solve this problem, in one embodiment, after determining the quantity statistical information of the target object based on the position distribution between the movement direction of the target object and the reference line set for the target area, it will also be determined whether there is an intersection between the movement direction corresponding to the counted target object and the first sub-area, which is a sub-area of ​​the target area; if it is determined that there is no intersection between the movement direction corresponding to the counted target object and the first sub-area, the quantity statistical information is changed.

[0104] Among them, the first sub-area can be set on the side close to the aisle exit. After the target object's number statistics are generated according to the target object "passing" the reference line, it will also be determined whether the target object corresponding to the recorded identity identifier is in or has passed through the first sub-area. If it is determined that the target object corresponding to the recorded identity identifier is not in or has not passed through the first sub-area, the number statistics value will be changed. Through the dual statistics of the first sub-area and the reference line, the number of target objects can be mutually checked, avoiding the problem of target number statistics errors caused by the complex design of the target area, thereby further improving the counting accuracy.

[0105] The relevant technology only monitors the quantity of animals during transportation, and does not effectively monitor or resolve personnel fraud.

[0106] To solve this problem, in one embodiment, the target objects include animals and people. After acquiring at least one frame of video image obtained by shooting the target area, it will also be detected whether there are animals in the second sub-area, and whether there are people in the third sub-area associated with the second sub-area. The second sub-area is a sub-area of ​​the target area, and the third sub-area is a sub-area of ​​the target area. When animals are detected in the second sub-area and people are detected in the third sub-area, an alarm message is issued.

[0107] Figure 7 A schematic diagram of the relative position relationship between the second sub-region and the third sub-region is given, wherein the third sub-region (B) is arranged on one of the two sides of the second sub-region (A). Figure 8 Another schematic diagram of the relative position relationship between the second sub-region and the third sub-region is given, wherein the third sub-region (B) is arranged around the second sub-region (A).

[0108] For example, Fig. 9 A schematic diagram of human intrusion alarm is given, where A is the second sub-area, such as the aisle, B is the third sub-area, such as the weighing area, and C is the alarm target. When pigs are transported, if a person appears in the third sub-area and stays there for a long time, an alarm message will be pushed. When pigs are not in the second sub-area and only people appear in the third sub-area, no alarm will be generated. This design can avoid false alarms when people are working in compliance with regulations.

[0109] In order to further avoid false alarms, a smoothing algorithm can be set up to use the personnel position obtained by detection and the personnel number obtained by tracking. The alarm will be triggered only when the same person stays in the third sub-area for more than a certain period of time. This time can be configured according to the actual situation. When the stay time is short, it will not be recorded. This can avoid false alarms caused by simple operations such as driving animals, and also avoid false alarms when the detection algorithm generates false detections. After the alarm information is generated, it can be matched with the pre-set access control algorithm (such as face recognition algorithm or work badge recognition algorithm). After confirming the identity information of the violator, it will be pushed to the enterprise information platform together so that the administrator can discover and deal with it in time.

[0110] In one embodiment, Fig.10 As shown, a neural network training method is provided. The neural network trained by the method can be applied to the above target detection scenario. The method includes the following steps:

[0111] Step S301, obtaining at least one frame of video image obtained by shooting the target area;

[0112] Step S302, using at least one frame of video image as a training sample to train the neural network to be trained, including: cropping at least part of the hidden layer in the neural network during training according to a reference ratio, and the reference ratio is determined based on the ratio of the size of the target object in the video image to the size of the entire video image.

[0113] In this embodiment, the target area can be an area in an animal transfer inventory scene, or an area in a personnel flow detection scene. The reference ratio refers to the size of the target object in the video image as a percentage of the size of the entire video image. Cropping some hidden layers divided in the neural network according to the reference ratio refers to mapping the reference ratio and the depth relationship of the hidden layer to be cropped. The larger the reference ratio, the deeper the hidden layer to be cropped. For example, animals such as pigs, cattle, sheep, horses, and people are medium to large targets in actual scenes, and do not require too many network layers for target detection. The neural network can be cropped along the depth direction to remove the deep network.

[0114] Among them, taking the object definition of COCO, a general dataset in the field of object detection, as an example, if the image size is 640*480, a small target refers to a target smaller than 32*32 pixels, a medium target refers to a target between 32*32 and 96*96 pixels, and a large target refers to a target larger than 96*96 pixels. Taking animal images as an example, the resolution is about 1920*1080 pixels, which can be converted to a target size definition that is roughly three times that of COCO.

[0115] The depth of the network is determined by dividing the hidden layers in the neural network according to the distance between each hidden layer and the input image. Assuming that the neural network used in this embodiment has 53 layers, it can be considered that the one-third close to the input image is a shallow network, the middle one-third is an intermediate network, and the one-third close to the output image is a deep network.

[0116] In this embodiment, the neural network includes many stacked residual structure sub-networks. Among these structures, the shallow network features have high resolution and retain more complete original image information, which is suitable for extracting small targets. The deep network features have low resolution and are suitable for extracting large targets. The target object of this embodiment is a medium to large target in the actual scene. There is no need for too many network layers for target detection. The neural network can be pruned along the depth direction to remove the deep network. By pruning the neural network, the memory usage and time consumption are greatly reduced while the detection accuracy is not reduced, so that it can run on the chip of the front-end device and realize real-time target detection to meet the real-time requirements of target object statistics, which is convenient for migration and promotion in different scenarios.

[0117] In one embodiment, a neural network includes a first network, the first network includes an input end and an output end, and the first network is configured to extract image features of a target object in a video image; cropping at least part of the hidden layers in the neural network during training according to a reference ratio includes: dividing multiple hidden layers in the first network into levels according to the distance between each hidden layer and the input end; and deleting at least part of the hidden layers divided in the first network according to the relationship between the ratio and the division level.

[0118] In this embodiment, the first network includes many stacked residual structure sub-networks. Among these structures, the shallow network features have high resolution and retain more complete original image information, which is suitable for extracting small targets. The deep network features have low resolution and are suitable for extracting large targets.

[0119] In one embodiment, the first network also includes multiple channels; cropping at least part of the hidden layers in the neural network during training according to the reference ratio also includes: obtaining the contribution of the first image features extracted by each channel to the second image features output by the output end of the first network; and deleting channels whose contributions are lower than a first preset threshold.

[0120] In this embodiment, there is redundancy in the information stored in each channel of the first network. By obtaining the contribution of each channel to the extracted target feature and deleting the channels whose contribution is lower than the preset value, that is, retaining the channels with large amount of information and removing the channels with small amount of information, the weight sparse processing of the neural network is achieved.

[0121] Each network layer in the first network can be regarded as a cube with width, height and number of channels. The width and height can be regarded as the bottom of the cube, which is determined by the image sampling multiple and cannot be changed. The number of channels can be regarded as the thickness of the cube and can be changed. The number of channels can be set to 64, 128, 256 or 512. For networks with a large amount of information, the number of channels can be subtracted from 0 to 1 / 4, and for networks with a small amount of information, the number of channels can be subtracted from 1 / 2. Regarding the determination of the amount of information, for each network layer, the contribution of each network layer is reversely calculated from the final target feature to obtain the contribution of each network layer to the final amount of information. Assuming that the first network of this embodiment has 53 layers, it can be sorted according to the amount of information, and the network layers ranked in the last 1 / 3 to 1 / 2 can be trimmed.

[0122] In one embodiment, the first network also includes multiple residual structure sub-networks, each residual structure sub-network includes at least one channel; obtaining the contribution of the first image features extracted by each channel to the second image features includes: obtaining the information entropy of each residual structure sub-network, and determining the contribution of the first image features extracted by the channels of each residual structure sub-network to the second image features based on the information entropy.

[0123] In one embodiment, pruning at least part of the hidden layers in a neural network during training according to a reference ratio also includes: upsampling or downsampling multiple branch networks in the first network, wherein different branch networks after sampling have the same number of channels and the same number of hidden layers; and performing feature fusion on multiple branch networks based on the size of feature images extracted by each branch network to obtain a second network.

[0124] In this embodiment, multiple branch networks in the first network are up-sampled / down-sampled and then feature fused according to the principle that the obtained feature maps have the same size, so that shallow information and deep information are used at the same time, thereby obtaining more features. Specifically, a shallow network layer and a deep network layer are selected respectively (since the network layer is composed of multiple different repeated structures, the last layer of each repeated structure can be selected for fusion), and the width and height of the two are made consistent through upsampling or downsampling, and the number of channels is directly superimposed, which is equivalent to connecting the bottom surfaces of two cubes and stacking them together. Then, the features in the two cubes can be used in subsequent calculations, thereby increasing the number of features.

[0125] In one embodiment, the neural network further comprises a third network, wherein the third network is configured to output at least one of the type and position of the target object, wherein an input end of the third network is coupled to an output end of the second network.

[0126] In this embodiment, the third network is responsible for outputting the coordinate position of the target object and obtaining the category and position of the target object.

[0127] In one embodiment, training the neural network to be trained includes: adjusting network parameters of the neural network during training, and pruning at least part of the hidden layers of the neural network after the network parameters are adjusted according to a reference ratio to obtain the trained neural network; or, pruning at least part of the hidden layers of the neural network during training according to a reference ratio, and adjusting network parameters of the pruned neural network to obtain the trained neural network.

[0128] In this embodiment, by pruning the neural network during the training process, the accuracy of the neural network is not affected too much, while the complexity is kept as low as possible, thereby saving the memory space occupied by the neural network and shortening the calculation time.

[0129] In one embodiment, Fig.11 A schematic diagram of the structure of a front-end device is given, such as Fig.11 As shown, the front-end device includes: a data acquisition module and a front-end chip, wherein the data acquisition module is electrically connected to the front-end chip, and the data acquisition module can be a CCD / CMOS camera or an image acquisition card, which is used to convert the optical image into an analog / digital image and output it to the front-end chip for processing, and the front-end chip is used to execute the target object statistical method or target detection method in any of the above embodiments. The implementation solution provided by the front-end device to solve the problem is similar to the implementation solution recorded in the above method. For the specific definition of the front-end device, please refer to the definition of the object statistical method above, which will not be repeated here.

[0130] In one embodiment, taking animal transport inventory as an example, Fig.12 A schematic diagram of the internal architecture of a front-end device is given, such as Fig.12 As shown, a data acquisition module is deployed in the front-end device, and the front-end device can be mounted on the target area to capture video images of the target area and transmit the video images to the front-end chip for analysis.

[0131] The front-end chip is deployed with a target detection module, a target tracking module, a reference line counting module and an alarm module, wherein the data acquisition module is coupled to the data processing module, the target detection module is coupled to the target tracking module, and the target tracking module is coupled to the reference line counting module and the alarm module respectively.

[0132] The target detection module detects animals and / or people based on the deep learning algorithm to obtain the category and location of the target object. This embodiment adopts the Yolo algorithm, which prunes the neural network so that the algorithm can run in the front-end chip and realize real-time target detection to meet the real-time requirements of animal transportation while maintaining the detection accuracy.

[0133] The target tracking module, based on the KCF (Kernelized Correlation Filters) algorithm, extracts image features of the target detected by the target detection module and the preset area around it, including HOG features and grayscale features, and uses the extracted features to estimate the possible position of the target in the next frame image, and tracks the target by determining the next frame position. Among them, the target tracking module will generate a unique identity for each different target object for subsequent distinction.

[0134] The reference line counting module is used to count the animals tracked by the target tracking module. A line perpendicular to the direction of animal movement is drawn in the weighing area to be counted as a reference line. The position relationship between the animal center point and the reference line is compared in the direction of animal movement. When the animal center point exceeds this reference line, the number of animals counted increases by 1. When all animals are transferred, the statistical result is the total number of animals.

[0135] The alarm module is used to alarm the human target tracked by the target tracking module. First, a preset area that cannot be entered by human beings is demarcated, such as the weighing area. When the animal is running through the weighing area, if the target tracking module tracks the entry of a human being, and the human being stays for more than a certain time (such as a few seconds), an alarm signal is triggered, which is automatically pushed to the enterprise information platform so that the back-end supervisor can find and handle it in time. At the same time, this module can be used together with face recognition or work badge recognition solutions to determine the identity information of the person, so as to facilitate timely processing.

[0136] Each module in the above-mentioned front-end device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0137] In combination with the front-end device of the above embodiment, a target object statistics system is provided in this embodiment, including a back-end device and a front-end device, the back-end device is connected to the front-end device, and the back-end device is provided with an interactive interface, the interactive interface is used to receive information input by the user, and receive and output information generated by the front-end device. Among them, the back-end device can be a computer or a server, and real-time display is performed through a visual interactive interface.

[0138] In one embodiment, taking animal transport inventory as an example, Fig.13 is a schematic diagram of the animal transport statistics system of this embodiment. Fig.12The front-end device shown is connected to the enterprise information platform. The enterprise information platform is deployed in the back-end device and is used to control the operation of the front-end device and receive information transmitted by the front-end device, so that the administrator can timely receive the video data, animal population statistics and personnel fraud alarm information reported by the front-end device.

[0139] Based on the same inventive concept, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the target object statistics method, the steps of the target detection method, or the steps of the neural network training method of any of the above embodiments are implemented.

[0140] In one embodiment, the internal structure of the computer device can be as follows Fig.14 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an object statistics method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0141] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the target object statistics method, the target detection method, or the neural network training method in any of the above embodiments.

[0143] In one embodiment, a computer program product is provided, including a computer program, which is used by a processor to execute the steps of the target object statistics method, the target detection method, or the neural network training method in any of the above embodiments.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A target object statistical method, characterized in that: The method comprises: Acquire at least one frame of video image obtained by shooting a target area; the target area includes a corridor for transferring a target object from one place to another; Performing target detection on the video image to identify at least one target object; Tracking the target object, determining the moving direction of the target object, and generating an identity corresponding to the target object; Determining quantity statistics information of the target object according to a movement direction of the target object and a position distribution between reference lines set for the target area; Performing target detection on the video image to identify at least one target object includes: inputting the video image into a trained neural network to identify at least one of the type and position of the target object; After determining the quantity statistical information of the target object according to the position distribution between the movement direction of the target object and the reference line set for the target area, the method further includes: judging whether there is an intersection between the movement direction corresponding to the counted target object and a first sub-area, the first sub-area being a sub-area of ​​the target area; the first sub-area is set at one side of the aisle exit; if it is judged that there is no intersection between the movement direction corresponding to the counted target object and the first sub-area, changing the quantity statistical information; The target objects include animals and people. After acquiring at least one frame of video image obtained by shooting the target area, the method further includes: detecting whether there are animals in the second sub-area, and detecting whether there are people in the third sub-area associated with the second sub-area, the second sub-area being a sub-area of ​​the target area, and the third sub-area being a sub-area of ​​the target area; when animals are detected in the second sub-area and people are detected in the third sub-area, an alarm message is issued; the second sub-area includes an area corresponding to the aisle; the third sub-area is set on one of the two sides of the second sub-area, or the third sub-area is surrounded by the second sub-area.

2. The target object counting method according to claim 1, characterized in that: The trained neural network is obtained by cropping at least part of the hidden layer in the neural network during training according to a reference ratio, and the reference ratio is determined based on the ratio of the size of the target object in the video image to the size of the entire video image.

3. The target object counting method according to claim 1 or 2, characterized in that: The quantity statistical information includes quantity statistical values; Determining the quantity statistical information of the target object according to the movement direction of the target object and the position distribution between the reference lines set for the target area includes: Determine whether there is a first target object among the at least one target object, wherein the first target object includes a target object whose motion direction intersects the reference line; In response to the presence of a first target object in the at least one target object, the quantity statistical value is determined based on the identity identifier included in the counted target object identifier set.

4. The target object counting method according to claim 3, characterized in that: The target objects include N, where N is a natural number; Determining the quantity statistical value based on the identity identifier included in the counted target object identifier set includes: performing N counting operations in sequence to obtain the quantity statistical value obtained by the last counting operation, wherein the i+1th counting operation in the N counting operations includes: Selecting a first target object on which the counting operation is not performed from the N first target objects; In response to the identity identifier of the selected first target object being included in the counted target object identification set, the quantity statistical value obtained by the i-th counting operation is determined as the quantity statistical value obtained by the i+1-th counting operation; or, in response to the identity identifier of the unselected first target object being included in the counted target object identification set, the sum of the quantity statistical value obtained by the i-th counting operation and a preset value is determined as the quantity statistical value obtained by the i+1-th counting operation.

5. The target object counting method according to claim 3, characterized in that: The outer contour of the target object is provided with a target detection frame, and the first target object includes a target object whose motion direction intersects the reference line and whose center position of the target detection frame is located on a preset side of the reference line.

6. The target object counting method according to claim 1 or 2, characterized in that: The target area includes an entrance and an exit. In the video image, The direction from the entrance to the exit of the target area is not parallel to the reference line; and / or, The reference line passes through a preset position of the target area.

7. A neural network training method, characterized in that: For training a trained neural network according to any one of claims 1 to 6, the method comprising: Acquire at least one frame of video image obtained by shooting the target area; The at least one frame of video image is used as a training sample to train the neural network to be trained, including: cropping at least part of the hidden layer in the neural network during the training process according to a reference ratio, and the reference ratio is determined based on the ratio of the size of the target object in the video image to the size of the entire video image.

8. The neural network training method according to claim 7, characterized in that: The neural network includes a first network, the first network includes an input end and an output end, and the first network is configured to extract image features of a target object in the video image; Pruning at least part of the hidden layers of the neural network during training according to the reference ratio includes: Dividing the multiple hidden layers in the first network into levels according to the distance between each hidden layer and the input end; At least part of the hidden layers divided in the first network is deleted according to the relationship between the ratio and the division level.

9. The neural network training method according to claim 8, characterized in that: The first network also includes a plurality of channels; Pruning at least some of the hidden layers of the neural network during training according to the reference ratio also includes: Obtaining a contribution of the first image feature extracted by each channel to the second image feature output by the output end of the first network; Delete channels whose contribution is lower than a first preset threshold.

10. The neural network training method according to claim 9, characterized in that: The first network further includes a plurality of residual structure sub-networks, each of which includes at least one channel; and obtaining the contribution of the first image feature extracted by each channel to the second image feature includes: The information entropy of each residual structure sub-network is obtained, and the contribution of the first image feature extracted by the channel of each residual structure sub-network to the formation of the second image feature is determined according to the information entropy.

11. The neural network training method according to claim 8, characterized in that: Pruning at least some of the hidden layers of the neural network during training according to the reference ratio also includes: Performing upsampling or downsampling on the multiple branch networks in the first network, wherein the different branch networks after the sampling process have the same number of channels and the same number of hidden layers; According to the size of the feature image extracted by each branch network, the multiple branch networks are feature fused to obtain a second network.

12. The neural network training method according to claim 11, characterized in that: The neural network also includes a third network, characterized in that the third network is configured to output at least one of the type and position of the target object, wherein an input end of the third network is coupled to an output end of the second network.

13. The neural network training method according to any one of claims 7 to 12, characterized in that: Training the neural network to be trained includes: Adjusting the network parameters of the neural network during training, and pruning at least part of the hidden layers of the neural network after the network parameters are adjusted according to a reference ratio to obtain a trained neural network; or, At least part of the hidden layers in the neural network during training are pruned according to a reference ratio, and network parameters of the pruned neural network are adjusted to obtain a trained neural network.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 or 7 to 13 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 or 7 to 13 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 or 7 to 13 are implemented.

Images