Unsupervised learning top view head and shoulder angle acquisition method, system, medium and terminal

Abstract

The present invention provides a method, system, medium and terminal for obtaining head-shoulder angles from a top perspective using unsupervised learning. The method comprises the following steps: horizontally annotating the head and shoulders of a pedestrian in a target top perspective image with a human body frame to obtain a first top perspective image; obtaining an angled second top perspective image annotated with a human body frame based on the first top perspective image; inputting the second top perspective image into a rotating frame prediction network so that the rotating frame prediction network outputs a target pre-selected frame; processing the target pre-selected frame to obtain an angled target rotating frame corresponding to the head and shoulders of the pedestrian. The method for obtaining head-shoulder angles from a top perspective proposed by the present invention only requires manual angle annotation of a small number of human bodies in the top perspective image, and then, by training rotating human body frames and horizontal head-shoulder frames, rotating frame annotations of the head and shoulders can be generated through the rotating frame prediction network, thereby avoiding the tedious work of re-annotating dense head and shoulders and adjusting the angle.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian detection, and in particular relates to a method, system, medium and terminal for acquiring head-shoulder angles from a top viewing angle using unsupervised learning. Background Art

[0002] In pedestrian detection application scenarios, surveillance cameras installed from a top-view angle are widely used due to their wide detection range and low cost. However, pedestrian detection from a top-view angle is more prone to human body occlusion, and the parts of a person that are least likely to be blocked are often the head and shoulders. To meet the business needs of pedestrian counting and tracking, high detection of the head and shoulders becomes a necessary task for pedestrian detection from a top-view angle.

[0003] In the current mainstream deep learning target detection algorithms, the detection effect of the target object depends to a large extent on the feature learning of the convolutional neural network. The labeling box that fits the target object and the data set covering a wide range of scenes can enable the convolutional neural network to obtain better feature learning. In this top-view target detection task, since the size and posture of pedestrians will show large imaging differences at different positions, ordinary horizontal box annotations often frame most of the background information, cannot fit the detection subject, and cannot enable the convolutional neural network to obtain good feature learning. Therefore, in order to obtain high detection of heads and shoulders from a top perspective, a data set with comprehensive coverage of the scene and head and shoulder box angle annotations, that is, rotation box annotations, is required.

[0004] In the currently available public datasets, most pedestrians are labeled horizontally from a horizontal perspective, and there is no annotation of rotated frames from a top perspective. If the head and shoulders from a top perspective are manually annotated with rotated frames, adjusting the angle will bring greater annotation costs than horizontal detection due to the higher density of these two types of targets from a top perspective. The rotated frame annotation of the human body from a top perspective requires fewer annotations and is larger than the head and shoulders frame, making it easier to label. However, due to the irregular postures of pedestrians mentioned above (for example, the angles of the head and shoulders and the human body are not necessarily the same when not standing, and the human body and head and shoulders frames of the same person may not appear at the same time), it is not possible to directly design logic to deduce the rotated frames of the head and shoulders from the human body rotated frame. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a method, system, medium and terminal for obtaining head and shoulder angles from a top perspective using unsupervised learning, which is used to solve the problem in the prior art that the rotation frames of the human head and shoulders cannot be derived from the rotation frame of the human body.

[0006] To achieve the above-mentioned and other related purposes, the present invention provides an unsupervised learning method for obtaining head-shoulder angles from a top perspective, comprising the following steps: performing horizontal frame annotation on the head and shoulders of a pedestrian in a target top perspective image to obtain a first top perspective image; the target top perspective image includes at least one person; based on the first top perspective image, obtaining a second top perspective image with an angle after being annotated with a human body frame; inputting the second top perspective image into a rotating frame prediction network so that the rotating frame prediction network outputs a target pre-selected frame; processing the target pre-selected frame to obtain an angled target rotating frame corresponding to the head and shoulders of the pedestrian.

[0007] In one embodiment of the present invention, before the step of inputting the second top-view picture into the rotating box prediction network, the method further includes the following steps: building the rotating box prediction network; training the rotating box prediction network to obtain the trained rotating box prediction network; the step of inputting the second top-view picture into the rotating box prediction network includes the following steps: inputting the second top-view picture into the trained rotating box prediction network.

[0008] In one embodiment of the present invention, training the rotating box prediction network to obtain a trained rotating box prediction network includes the following steps: step 1, horizontally marking the heads and shoulders of pedestrians in a training top-view image to obtain a third top-view image; the training top-view image includes at least one pedestrian; step 2, based on the third top-view image, obtaining a fourth top-view image with an angle after human body frame marking; step 3, inputting the fourth top-view image into the rotating box prediction network so that the rotating box prediction network outputs a training pre-selected box; step 4, processing the training pre-selected box to obtain an angled training rotating box corresponding to the human body; step 5, calculating the difference between the angle in the training rotating box and the angle in the fourth top-view image to obtain an angle loss value, and calculating a total loss value based on the angle loss value; step 6, looping through steps 1 to 5 until the total loss value no longer decreases, thereby obtaining the trained rotating box prediction network.

[0009] In one embodiment of the present invention, the rotation box prediction network includes: a feature extraction module; the feature extraction module is used to output a feature map group corresponding to the pedestrian's body, head and shoulders respectively; each feature map group includes: a first feature map at a first scale, a second feature map at a second scale, and a third feature map at a third scale; the second scale is larger than the first scale, and the second scale is smaller than the third scale; all the first feature maps at each scale correspond to a feature matrix; before the step of inputting the second top-view image into the rotation box prediction network, the method further includes the following steps: designing the rotation box prediction network so that the rotation box prediction network strengthens the fusion of the third feature map and the first feature map in each feature map group; inputting the second top-view image into the rotation box prediction network includes the following steps: inputting the second top-view image into the designed rotation box prediction network.

[0010] In one embodiment of the present invention, before the step of inputting the second top-view image into the rotating box prediction network, the method further includes the following steps: clustering the head, shoulders, and body frames marked in the second top-view image respectively; configuring a pre-selection box selection strategy of the rotating box prediction network; the pre-selection box selection strategy is: the center point of the pre-selection box and four grids above, below, left, and right of the center point; inputting the second top-view image into the rotating box prediction network includes the following steps: inputting the clustered second top-view image into the configured rotating box prediction network.

[0011] In one embodiment of the present invention, the processing of the target pre-selection box includes the following steps: performing non-maximum suppression processing on the target pre-selection box according to categories; the categories include: human body, head, and shoulders.

[0012] In one embodiment of the present invention, the target pre-selection box includes: a target human body pre-selection box corresponding to the pedestrian's body, a target head pre-selection box corresponding to the head, and a target shoulder pre-selection box corresponding to the shoulder; obtaining the angled target rotation box corresponding to the pedestrian's head and shoulders includes the following steps: adding 180 channels based on the Yolov5 algorithm to predict the angle.

[0013] The present invention provides an unsupervised learning top-view head-shoulder angle acquisition system, comprising: a labeling module, an acquisition module, an input module, and a processing module; the labeling module is used to perform horizontal frame annotation on the head and shoulders of a pedestrian in a target top-view image to obtain a first top-view image; the target top-view image includes at least one pedestrian; the acquisition module is used to obtain, based on the first top-view image, a second top-view image with an angle after being annotated with a human body frame; the input module is used to input the second top-view image into a rotation frame prediction network, so that the rotation frame prediction network outputs a target pre-selection frame; and the processing module is used to process the target pre-selection frame to obtain an angled target rotation frame corresponding to the head and shoulders of the pedestrian.

[0014] The present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned unsupervised learning method for acquiring the head-shoulder angle from a top viewing angle.

[0015] The present invention provides a terminal, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the terminal performs the above-mentioned unsupervised learning top-view head and shoulder angle acquisition method.

[0016] As described above, the unsupervised learning method, system, medium, and terminal for acquiring head-shoulder angles from a top viewing angle have the following beneficial effects:

[0017] (1) Compared with the existing technology, the top view head and shoulder angle acquisition method proposed in the present invention only requires manual angle annotation of a small number of human bodies in the top view image. Then, by training the rotated human body frame and the horizontal head and shoulder frame, the rotated frame annotation of the head and shoulders can be generated through the rotated frame prediction network, avoiding the tedious work of re-annotating and adjusting the angles of dense head and shoulders.

[0018] (2) After training, the rotation frame prediction network proposed in the present invention can not only obtain the head and shoulder angles of the top-view image, but also mark three types of rotation frames (including human body, head and shoulders) for the top-view image.

[0019] (3) The rotating frame prediction network proposed in the present invention is trained by pre-selecting frames of the human body, head, and shoulders, which improves the feature recognition ability of the rotating frame prediction network for various parts of the human body, and can effectively improve the detection rate of top-view head and shoulder detection, thereby obtaining high-quality top-view head and shoulder detection annotations.

[0020] (4) The training method and rotating box prediction network proposed in the present invention select multiple detection frameworks (the backbone of Yolov3, SPP and PAN of Yolov4, and the pre-selected box and loss value of Yolov5) for business scenarios and combine and improve them, which can enable the rotating box prediction network to converge quickly and shorten the training process. The training method proposed in the present invention can also be applied to the training of small front-end models.

[0021] (5) The aspect ratio sensitive weight W proposed in this invention RSW This enables the rotation frame prediction network to better learn the human body angle features, so that the head and shoulders can also naturally learn the human body angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Shown is a schematic structural diagram of a terminal in one embodiment of the present invention.

[0023] Figure 2 FIG. 1 is a flow chart of an unsupervised learning top-view head-and-shoulder angle acquisition method according to an embodiment of the present invention.

[0024] Figure 3 FIG. 1 is a schematic diagram showing the structure of an SPP module in which a convolution replaces a pooling layer in one embodiment of the present invention.

[0025] Figure 4 FIG. 1 is a schematic diagram showing the structure of a rotated frame prediction network according to an embodiment of the present invention.

[0026] Figure 5 Shown is a flowchart of training a rotation box prediction network and obtaining a trained rotation box prediction network in one embodiment of the present invention.

[0027] Figure 6 The figure shows a comparison of the angle difference when w / r is large according to the present invention.

[0028] Figure 7 The figure shows a comparison of angle differences when w / r is close to 1 according to the present invention.

[0029] Figure 8 FIG. 1 is a schematic diagram showing the structure of an unsupervised learning top-view head-and-shoulder angle acquisition system in one embodiment of the present invention.

[0030] Explanation of symbols

[0031] 1 Terminal

[0032] 11 processing units

[0033] 12 Memory

[0034] 121 Random Access Memory

[0035] 122 Cache memory

[0036] 123 Storage System

[0037] 124 Programs / Utilities

[0038] 1241 Program Module

[0039] 13 Bus

[0040] 14 Input / Output Interfaces

[0041] 15 Network Adapter

[0042] 2 External devices

[0043] 3 Display

[0044] 81 Annotation Module

[0045] 82 Get Module

[0046] 83 Input Module

[0047] 84 Processing Module

[0048] Steps S1 to S4

[0049] Steps S121 to S126 DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0051] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0052] The unsupervised learning top-view head-shoulder angle acquisition method, system, medium and terminal of the present invention, compared with the existing technology, the top-view head-shoulder angle acquisition method proposed by the present invention only requires manual angle annotation of a small number of human bodies in the top-view image, and then by training the rotating human body frame and the horizontal head-shoulder frame, the rotating frame annotation of the head and shoulders can be generated by the rotating frame prediction network, avoiding the tedious work of re-annotating and adjusting the angle of dense head and shoulders; after training, the rotating frame prediction network proposed by the present invention can not only obtain the head and shoulder angles of the top-view image, but also annotate three types of rotating frames (including human body, head and shoulders) for the top-view image; the rotating frame prediction network proposed by the present invention can generate the rotating frame annotation of the head and shoulders by training the rotating human body frame and the horizontal head-shoulder frame, thereby avoiding the tedious work of re-annotating and adjusting the angle of dense head and shoulders; after training, the rotating frame prediction network proposed by the present invention can not only obtain the head and shoulder angles of the top-view image, but also annotate the rotating frames of three types (including human body, head and shoulders) for the top-view image; The prediction network is trained with pre-selected boxes of the human body, head, and shoulders, which improves the feature recognition ability of the rotating box prediction network for various parts of the human body, and can effectively improve the detection rate of top-view head and shoulder detection, thereby obtaining high-quality top-view head and shoulder detection annotations; the training method and rotating box prediction network proposed in the present invention select multiple detection frameworks (the backbone of Yolov3, SPP and PAN of Yolov4, and the pre-selected box and loss value of Yolov5) for business scenarios and combine and improve them, which can enable the rotating box prediction network to converge quickly and shorten the training process. The training method proposed in the present invention can also be applied to the training of small front-end models; the aspect ratio sensitive weight W proposed in the present invention is RSW This enables the rotation frame prediction network to better learn the human body angle features, so that the head and shoulders can also naturally learn the human body angle.

[0053] The storage medium of the present invention stores a computer program that, when executed by a processor, implements the following unsupervised learning method for acquiring head-and-shoulder angles from a top-view angle. The storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), a magnetic disk, a USB flash drive, a memory card, or an optical disk.

[0054] Any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0055] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0056] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0057] The computer program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0058] The present invention will be described below with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine such that when these computer program instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams.

[0059] These computer program instructions may also be stored in a computer-readable medium, which causes a computer, other programmable data processing apparatus, or other device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions for implementing the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0060] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0061] The terminal of the present invention includes a processor and a memory.

[0062] The memory is used to store computer programs; preferably, the memory includes: ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk and other media that can store program codes.

[0063] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the terminal performs the following unsupervised learning top view head and shoulder angle acquisition method.

[0064] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0065] Figure 1 A block diagram of an exemplary terminal 1 suitable for implementing embodiments of the present invention is shown.

[0066] Figure 1 The terminal 1 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0067] like Figure 1 As shown, the terminal 1 is in the form of a general-purpose computing device. Components of the terminal 1 may include, but are not limited to, one or more processors or processing units 11, a memory 12, and a bus 13 connecting different system components (including the memory 12 and the processing unit 11).

[0068] Bus 13 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of such architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0069] The terminal 1 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the terminal 1, including volatile and non-volatile media, removable and non-removable media.

[0070] The memory 12 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 121 and / or cache memory 122. The terminal 1 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 123 may be used to read and write non-removable, non-volatile magnetic media ( Figure 1 Not shown, often called a "hard drive"). Although Figure 1 Although not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 13 via one or more data medium interfaces. Memory 12 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0071] A program / utility 124 having a set (at least one) of program modules 1241 may be stored, for example, in memory 12. Such program modules 1241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 1241 generally implement the functions and / or methods of the embodiments described herein.

[0072] The terminal 1 may also communicate with one or more external devices 2 (e.g., a keyboard, a pointing device, a display 3, etc.), one or more devices that enable a user to interact with the terminal 1, and / or any device that enables the terminal 1 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 14. Furthermore, the terminal 1 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 15. Figure 1 As shown, the network adapter 15 communicates with other modules of the terminal 1 via the bus 13. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the terminal 1, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0073] like Figure 2 As shown, in one embodiment, the unsupervised learning top view head and shoulder angle acquisition method of the present invention includes the following steps:

[0074] Step S1: Mark the head and shoulders of the pedestrian in the target top-view image with a horizontal frame to obtain a first top-view image.

[0075] It should be noted that the target top-view image includes at least one group of people.

[0076] In one embodiment, the target top-view image is detected based on an existing open source detection framework or pedestrian detection interface to obtain horizontal frame annotations of the head and shoulders under the top view.

[0077] In one embodiment, in step S1, after being marked with a horizontal frame, the horizontal frame is as follows (x center ,y center ,w,h) description, where x cen t er ,y center is the coordinate of the center point of the horizontal frame; w and h are the width and height of the horizontal frame respectively.

[0078] Step S2: Based on the first top-view image, a second top-view image with an angle and annotated with a human body frame is obtained.

[0079] In one embodiment, the first top-view image is manually labeled with a human frame to obtain a second top-view image with an angled human frame.

[0080] It should be noted that the width of the above human body frame is defined as the long side of the rectangular frame, the height is defined as the short side of the rectangular frame, and the angle θ of the rectangular frame is defined as the angle between the long side and the y-axis, ranging from (0°, 180°).

[0081] Step S3: input the second top-view image into a rotating box prediction network, so that the rotating box prediction network outputs a target pre-selection box.

[0082] Specifically, the second top-view image obtained in step S2 is input into the rotating box prediction network, so that the rotating box prediction network outputs the target pre-selection box.

[0083] In one embodiment, the rotated frame prediction network includes a feature extraction module.

[0084] Specifically, the feature extraction module is used to output a group of feature maps corresponding to the pedestrian's body, head and shoulders respectively; each of the feature map groups includes: a first feature map at a first scale, a second feature map at a second scale and a third feature map at a third scale; wherein the second scale is larger than the first scale, and the second scale is smaller than the third scale; all the first feature maps at each scale correspond to a feature matrix.

[0085] In one embodiment, the feature extraction module uses Darknet53 in Yolov3 as the feature extraction backbone, and uses it to output feature maps of three scales (stride8, stride16, stride32).

[0086] In one embodiment, the target pre-selection frame includes: a target human body pre-selection frame corresponding to the human body of the pedestrian, a target head pre-selection frame corresponding to the head, and a target shoulder pre-selection frame corresponding to the shoulder.

[0087] Step S4: Process the target pre-selected frame to obtain an angled target rotation frame corresponding to the pedestrian's head and shoulders.

[0088] In one embodiment, the processing of the target pre-selection box includes the following steps: performing non-maximum suppression processing on the target pre-selection box according to categories.

[0089] It should be noted that non-maximum suppression (NMS), as the name suggests, suppresses elements that are not maximum values. When used in target detection, it extracts target detection frames with high confidence and suppresses false detection frames with low confidence. Generally speaking, when the parsing model outputs to the target frame, there will be a large number of target frames, the specific number of which is determined by the number of anchors. There are many duplicate frames that locate the same target. NMS is used to remove these duplicate frames and obtain the true target frame.

[0090] It should be noted that the categories include: human body, head, and shoulders.

[0091] In one embodiment, obtaining the angled target rotation frame corresponding to the pedestrian's head and shoulders includes the following steps: adding 180 channels based on the Yolov5 algorithm to predict the angle.

[0092] like Figure 4 As shown, an RGB image with width w and height h is input into the rotation box prediction network, and finally a feature matrix of stride8 (w / 8, h / 8, out_chn), stride16 (w / 16, h / 16, out_chn), and stride32 (w / 32, h / 32, out_chn) is obtained. In one embodiment, the processing method after step S3 is similar to Yolov5, with the following differences:

[0093] The number of output channels in this embodiment is:

[0094] out_chn=(x center ,y center ,w,h,conf,category)×3(number of anchors)+180(angle prediction);

[0095] Compared with Yolov5, this embodiment adds 180 channels to predict angles.

[0096] It should be noted that 180 channels are set to smooth the angle and avoid the angle periodicity problem. The number of channels with the maximum value after the sigmoid function calculation represents the angle of the pre-selected box. For the feature map with a width and height of (w / stride, h / stride), the network prediction can obtain 3×(w / stride)×(h / stride) boxes (x center ,y center ,w,h,conf,category), the angles of the boxes of these three anchors are all determined by 180 angle channels. For a certain point on the feature matrix, the angle of the pre-selected box obtained by it is the same, that is, no matter which category is finally predicted or which anchor is selected, its angle is determined by the receptive field of a certain point on the feature matrix. In this way, during the training process, the network can obtain the ability to extract angle features from the receptive field through training on the human body angle, and migrate it to the angle prediction of the head and shoulders.

[0097] Furthermore, corresponding to the above setting of 180 channels to predict the angle, the setting of pre-selection box is added to predict x center ,y center The channel encoding method is different from Yolov5's sigmod(x)×2-0.5+grid bias, and is changed to sigmod(x)×3-1.5+grid bias.

[0098] In one embodiment, before the step of inputting the second top-view image into the rotated frame prediction network, the method further includes the following steps:

[0099] (11) Build the rotation frame prediction network.

[0100] (12) Training the rotating frame prediction network to obtain a trained rotating frame prediction network.

[0101] It should be noted that in the first training, the training data (corresponding to the top-view training image below) does not contain shoulder and head angle values. The network needs to rely on the learning of human body angles to learn the angles of these parts. The prediction of the head and shoulders mainly relies on small-scale feature maps, and the prediction of the human body mainly relies on large-scale feature maps. Therefore, the network needs to strengthen the fusion of large-scale feature maps (stride32) and small-scale feature maps (stride8).

[0102] In one embodiment, before the step of inputting the second top-view image into the rotated box prediction network, the method further includes the following steps: designing the rotated box prediction network so that the rotated box prediction network strengthens the fusion of the third feature map and the first feature map in each feature map group.

[0103] like Figure 3 As shown, in one embodiment, a method similar to Yolov4 is adopted, and a method similar to SPP is added before the feature map of Stride32, and a convolution layer is used instead of a pooling layer, which can better fuse multiple scale features.

[0104] like Figure 4 As shown in the figure, the FPN in the Yolov3 detection head is changed to PAN. The path in the figure shows that the thick dashed line path added in the PAN structure passes through fewer convolutional layers than the thin dashed line, shortening the path of large-scale layer fusion of small-scale layers, making it easier for the loss of large-scale layers to be propagated back to small-scale layers. On the one hand, it enables the small-scale layer to learn large-scale information, and on the other hand, it enables the large-scale layer to fuse more small-scale information.

[0105] It should be noted that the rotation box prediction network uses Darknet53 combined with convolution to replace the SPP and PAN output layers of the pooling layer, so that the representation of the three targets of different scales, namely the head, shoulders and human body, in the feature layer is integrated and enhanced; by strengthening the feature fusion of feature layers of different scales through the network, the network backbone can simultaneously receive information from receptive fields of different scales, thereby strengthening the information interconnection between different parts.

[0106] In one embodiment, the step of inputting the second top-view image into the rotation box prediction network comprises the following steps: inputting the second top-view image into the designed rotation box prediction network.

[0107] like Figure 5 As shown, in one embodiment, the training of the rotating frame prediction network and obtaining the trained rotating frame prediction network includes the following steps:

[0108] Step S121: perform horizontal frame annotation on the head and shoulders of the pedestrian in the training top-view image to obtain a third top-view image.

[0109] It should be noted that the working principle of step S121 is the same as that of the above-mentioned step S1, so it will not be described in detail here.

[0110] It should be noted that the training top-view picture includes at least one row of people.

[0111] Step S122: Based on the third top-view image, a fourth top-view image with an angle and annotated with a human body frame is obtained.

[0112] It should be noted that the working principle of step S122 is the same as that of the above-mentioned step S2, so it will not be described in detail here.

[0113] Step S123: input the fourth top-view image into the rotating box prediction network, so that the rotating box prediction network outputs a training pre-selection box.

[0114] It should be noted that the working principle of step S123 is the same as that of the above-mentioned step S3, so it will not be described in detail here.

[0115] Step S124: Process the training pre-selected frame to obtain an angled training rotation frame corresponding to the human body.

[0116] It should be noted that the working principle of step S124 is the same as that of the above-mentioned step S4, so it will not be described in detail here.

[0117] Step S125: Calculate the difference between the angle in the training rotation frame and the angle in the fourth top perspective image to obtain an angle loss value, and calculate a total loss value based on the angle loss value.

[0118] In one embodiment, the total loss value is the sum of the loss values ​​of the three feature matrices. The loss value of the li-th feature matrix can be calculated as follows:

[0119]

[0120] Among them, Loss 1i represents the loss value of the ith feature matrix output by the rotating frame prediction network; S 2 =(w / stride)×(h / stride) represents the number of grids in the feature map; Indicates that the calculation range includes all grids in this layer; nproposal indicates that 2 The number of pre-selected boxes selected in the grid; Indicates that the calculation range is all the training pre-selected boxes.

[0121] It should be noted that the first term in the above loss value calculation formula is the confidence loss, which calculates the prediction loss of all grids; target When the grid is a pre-selected box, it is the IOU value between it and the gt box, and when the grid is not a pre-selected box, it is 0; the second term in the above loss value calculation formula is the box coordinate loss; coordinate target ,coordinate predict The center coordinates and width and height (x, y, w, h) of the target box and pre-selected box respectively; L CIOUis the CIOU loss function; CIOU can better reflect the fit of the two frames. It adds coverage area, center point distance, and aspect ratio considerations on the basis of IOU; the third term in the above loss value calculation formula is the category loss of the frame; L cross-entropy is the cross entropy loss function; the fourth term in the above loss value calculation formula is the angle loss. In this step, only the angle loss of the target box and the pre-selected box of the human category is calculated (because there is no angle annotation for the head and shoulders); L focalloss They are Focal Loss loss functions, which calculate the 180-dimensional angle vector Θ in the pre-selected box. predict The target value vector Θ smoothed with the angle value target The loss of W is designed here. RSW (Aspect ratio sensitive weight) adjusts the Focal Loss angle loss. The principle is as follows:

[0122] When the r=w / h value is large, the difference between the predicted value and the target value is close to 90°, e.g. Figure 6 As shown, when the angle difference predict2>predict1>predict3, the difference between them is predict1>predict2=predict3; Figure 7 As shown in the figure, as the value of r = w / h decreases to 1, the difference becomes less obvious, and the most obvious difference is at 45°.

[0123] W RSW (Δθ)=|sin(Δθ+arccot(r 2 ))|:

[0124] Where r = w target / h target ; Δθ=θ target -θ predict ;W RSW The phase shift of the sin function to the angle difference will be adjusted according to r; when w / h is large, W RSW When the angle difference is 90°, it reaches the maximum value. As w / h approaches 1, W RSW The maximum value of the left shift; when w / h=1, W RSW When the angle difference is 45°, it reaches the maximum value; RSW The angle error can be smoothly adjusted according to the change of r, effectively describing the prediction difference caused by the angle error under different r conditions.

[0125] Step S126: loop through steps S121 to S125 until the total loss value stops decreasing, and obtain the trained rotation box prediction network.

[0126] Further, when the total loss value no longer decreases, a weight file is obtained.

[0127] It should be noted that all pre-selected boxes share the feature extraction weights and a set of angle predictions at the same feature point, so that the rotation box prediction network actually learns the ability to obtain angle information from the receptive field.

[0128] In one embodiment, inputting the second top-view image into the rotation box prediction network comprises the following steps: inputting the second top-view image into the trained rotation box prediction network.

[0129] In one embodiment, before the step of inputting the second top-view image into the rotated frame prediction network, the method further includes the following steps: clustering the head, shoulder, and body frames marked in the second top-view image respectively.

[0130] It should be noted that due to the wide range of top view scales and the obvious distinction in aspect ratios of the head, shoulders, and human body frames, there should be anchors of different scales for each category.

[0131] In one embodiment, kmeans clustering is performed on the head, shoulder, and body frames in the annotations, and the number of cluster categories is set to 3. Anchors of three scales are obtained for each category frame, and they are distributed to three feature layers according to the scales.

[0132] It should be noted that the above-mentioned anchor design enables the head, head-shoulder, and human body to have corresponding anchors at different feature layers, which can effectively improve the detection rate.

[0133] In one embodiment, before the step of inputting the second top-view image into the rotating box prediction network, the method further includes the following step: configuring a pre-selection box selection strategy of the rotating box prediction network.

[0134] In one embodiment, the pre-selection box selection strategy is: the center point of the pre-selection box and four grids above, below, left and right of the center point.

[0135] It should be noted that due to the large scale variation of the target object under the top perspective, the scale of a target box may be between the three feature layers, that is, both the large-scale feature layer and the small-scale feature layer may predict the box, so a gt box should be matched by multiple layers, that is, the gt box can be matched with the anchor of each feature layer, and if it is matched, it becomes the target that the layer is responsible for predicting; the pre-selected box generation strategy of yolov5 meets this requirement, and in order to improve the detection rate of the box, this example is changed to its center point and its four upper, lower, left and right grids (the original yolov5 pre-selected box selection strategy is to select the mapping grid of the center point on the feature map and the two grids close to it), that is, 1 target box on the same feature layer will have 5 pre-selected boxes and calculate the loss with it.

[0136] In one embodiment, inputting the second top-view image into the rotation box prediction network comprises the following steps: inputting the clustered second top-view image into the configured rotation box prediction network.

[0137] It should be noted that the head and shoulders are local parts of the human body, and their angles are strongly correlated with the angles of the human body. The present invention allows the rotating frame prediction network to learn the position and angle information of the human body while learning the position information of the head and shoulders. This allows the rotating frame prediction network to utilize the information fed back from various parts to extract human features. That is, in the process of learning to predict the angles and positions of the human body, the rotating frame prediction network needs to integrate information from local parts of the human body (head and shoulders) to make judgments.

[0138] Furthermore, the present invention enables the rotation frame prediction network to transfer the prediction of human body angles to the head and shoulders; the present invention is an improvement on the existing detection technology solution, realizing unsupervised learning of top-view head and shoulder angles, reducing the burden of manual labeling.

[0139] It should be noted that when it is necessary to obtain the head and shoulder angle of a pedestrian in a top view picture to be measured, the top view picture to be measured is input into the above-mentioned rotating frame prediction network, and then the feature maps of the three strides output by the rotating frame prediction network are processed accordingly to obtain The angles corresponding to different anchors on the same feature point are the same. By performing NMS processing on these frames, we can obtain three types of rotation frames: head, shoulder, and human body.

[0140] It should be noted that the protection scope of the unsupervised learning top-view head and shoulder angle acquisition method described in the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing, or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0141] like Figure 8As shown, in one embodiment, the unsupervised learning top view head and shoulder angle acquisition system of the present invention includes a labeling module 81 , an acquisition module 82 , an input module 83 and a processing module 84 .

[0142] The labeling module 81 is used to perform horizontal frame labeling on the heads and shoulders of pedestrians in the target top-view image to obtain a first top-view image; the target top-view image includes at least one person.

[0143] The acquisition module 82 is used to acquire an angled second top-view image marked with a human body frame based on the first top-view image.

[0144] The input module 83 is configured to input the second top-view image into a rotating box prediction network, so that the rotating box prediction network outputs a target pre-selection box.

[0145] The processing module 84 is configured to process the target pre-selected frame and obtain an angled target rotation frame corresponding to the pedestrian's head and shoulders.

[0146] It should be noted that the structures and principles of the labeling module 81, the acquisition module 82, the input module 83 and the processing module 84 correspond one-to-one to the steps (steps S1 to S4) in the above-mentioned unsupervised learning top-view head and shoulder angle acquisition method, so they are not repeated here.

[0147] It should be understood that the division of the modules in the above system is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules may be implemented entirely in software called by a processing element, or entirely in hardware. Alternatively, some modules may be implemented in software called by a processing element, while others may be implemented in hardware. For example, module x may be a separate processing element, or integrated into a chip in the above system. Furthermore, it may be stored in the form of program code in the memory of the above system, called by a processing element in the system to perform the functions of module x. The implementation of other modules is similar. Furthermore, these modules may be fully or partially integrated or implemented independently. The processing element described herein may be an integrated circuit with signal processing capabilities. During implementation, the steps of the above method or the modules above may be performed by hardware integrated logic circuits in the processor element or by software instructions.

[0148] For example, the above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0149] It should be noted that the unsupervised learning top-view head-and-shoulder angle acquisition system of the present invention can implement the unsupervised learning top-view head-and-shoulder angle acquisition method of the present invention, but the implementation device of the unsupervised learning top-view head-and-shoulder angle acquisition method of the present invention includes but is not limited to the structure of the unsupervised learning top-view head-and-shoulder angle acquisition system listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.

[0150] In summary, the unsupervised learning top-view head-shoulder angle acquisition method, system, medium and terminal of the present invention, compared with the prior art, the top-view head-shoulder angle acquisition method proposed by the present invention only requires manual angle annotation of a small number of human bodies in the top-view picture, and then by training the rotated human body frame and the horizontal head-shoulder frame, the rotated frame annotation of the head and shoulders can be generated by the rotated frame prediction network, avoiding the tedious work of re-annotating and adjusting the angles of dense head and shoulders; after training, the rotated frame prediction network proposed by the present invention can not only obtain the head and shoulder angles of the top-view picture, but also annotate three types of rotated frames (including human body, head and shoulders) for the top-view picture; the rotated frame prediction network proposed by the present invention can generate the head-shoulder angles of the head and shoulders of the top-view picture, and can also annotate the three types of rotated frames (including human body, head and shoulders) for the top-view picture. The rotation box prediction network is trained with pre-selected boxes of the human body, head, and shoulders, which improves the feature recognition ability of the rotation box prediction network for various parts of the human body, and can effectively improve the detection rate of top-view head and shoulder detection, thereby obtaining high-quality top-view head and shoulder detection annotations; the training method and rotation box prediction network proposed in the present invention select multiple detection frameworks (the backbone of Yolov3, SPP and PAN of Yolov4, and the pre-selected box and loss value of Yolov5) for business scenarios and combine and improve them, which can enable the rotation box prediction network to converge quickly and shorten the training process. The training method proposed in the present invention can also be applied to the training of small front-end models; the aspect ratio sensitive weight W proposed in the present invention is RSW The rotation frame prediction network can better learn the angle features of the human body, so that the head and shoulders can also naturally learn the angles of the human body; therefore, the present invention effectively overcomes the various shortcomings of the existing technology and has high industrial utilization value.

[0151] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. An unsupervised learning method for obtaining head and shoulder angles from a top view, characterized in that: The following steps are involved: Marking the heads and shoulders of pedestrians in a target top-view image with horizontal frames to obtain a first top-view image; the target top-view image includes at least one person; Based on the first top-view image, obtain a second top-view image with an angle and annotated with a human body frame; Inputting the second top-view picture into a rotating box prediction network so that the rotating box prediction network outputs a target pre-selection box; before the step of inputting the second top-view picture into the rotating box prediction network; building the rotating box prediction network; Training the rotating frame prediction network to obtain a trained rotating frame prediction network; The step of inputting the second top-view picture into the rotating frame prediction network comprises the following steps: inputting the second top-view picture into the trained rotating frame prediction network; and the step of training the rotating frame prediction network to obtain the trained rotating frame prediction network comprises the following steps: Step 1: Mark the heads and shoulders of pedestrians in the training top-view image with horizontal frames to obtain a third top-view image; the training top-view image includes at least one person; Step 2: obtaining a fourth top-view image with an angle after being marked with a human body frame based on the third top-view image; Step 3: inputting the fourth top-view image into the rotating frame prediction network, so that the rotating frame prediction network outputs a training pre-selection frame; Step 4: Process the training pre-selected frame to obtain an angled training rotation frame corresponding to the human body; Step 5: Calculate the difference between the angle in the training rotation frame and the angle in the fourth top view image to obtain an angle loss value, and calculate the total loss value based on the angle loss value; Step 6: looping through steps 1 to 5 until the total loss value stops decreasing, and obtaining the trained rotation frame prediction network; The target pre-selected frame is processed to obtain an angled target rotation frame corresponding to the pedestrian's head and shoulders.

2. The unsupervised learning method for acquiring head-shoulder angles from top viewing angles according to claim 1, characterized in that: The rotation frame prediction network includes: a feature extraction module; The feature extraction module is configured to output a set of feature maps corresponding to the body, head, and shoulders of the pedestrian, respectively; each set of feature maps includes: a first feature map at a first scale, a second feature map at a second scale, and a third feature map at a third scale; the second scale is larger than the first scale, and the second scale is smaller than the third scale; all the first feature maps at each scale correspond to a feature matrix; Before the step of inputting the second top-view image into the rotation frame prediction network, the method further includes the following steps: designing the rotation frame prediction network so that the rotation frame prediction network strengthens the fusion of the third feature map and the first feature map in each feature map group; The step of inputting the second top-view picture into the rotating frame prediction network includes the following steps: inputting the second top-view picture into the designed rotating frame prediction network.

3. The unsupervised learning method for acquiring head-shoulder angles from top viewing angles according to claim 1, characterized in that: Before the step of inputting the second top-view picture into the rotated frame prediction network, the method further includes the following steps: Clustering the head, shoulders, and body frames marked in the second top-view image respectively; Configuring a pre-selection box selection strategy for the rotating box prediction network; the pre-selection box selection strategy is: a center point of the pre-selection box and four grids above, below, left, and right of the center point; Inputting the second top-view image into the rotating frame prediction network includes the following steps: inputting the clustered second top-view image into the configured rotating frame prediction network.

4. The unsupervised learning method for acquiring head-shoulder angles from top viewing angles according to claim 1, characterized in that: The processing of the target pre-selection box includes the following steps: performing non-maximum suppression processing on the target pre-selection box according to categories; the categories include: human body, head, and shoulders.

5. The unsupervised learning method for acquiring head-shoulder angles from top viewing angles according to claim 1, characterized in that: The target pre-selection frame includes: a target human body pre-selection frame corresponding to the human body of the pedestrian, a target head pre-selection frame corresponding to the head, and a target shoulder pre-selection frame corresponding to the shoulder; The obtaining of the target rotating frame with angles corresponding to the pedestrian's head and shoulders includes the following steps: adding 180 channels based on the Yolov5 algorithm to predict the angles.

6. An unsupervised learning system for acquiring head and shoulder angles from top view, characterized by: include: Annotation module, acquisition module, input module and processing module; The annotation module is used to perform horizontal frame annotation on the head and shoulders of pedestrians in the target top-view image to obtain a first top-view image; the target top-view image includes at least one person; The acquisition module is used to acquire a second top-view image with an angle after being marked with a human body frame based on the first top-view image; The input module is used to input the second top-view picture into the rotating box prediction network, so that the rotating box prediction network outputs a target pre-selection box; before the step of inputting the second top-view picture into the rotating box prediction network; building the rotating box prediction network; Training the rotating frame prediction network to obtain a trained rotating frame prediction network; The step of inputting the second top-view picture into the rotating frame prediction network comprises the following steps: inputting the second top-view picture into the trained rotating frame prediction network; and the step of training the rotating frame prediction network to obtain the trained rotating frame prediction network comprises the following steps: Step 1: Mark the heads and shoulders of pedestrians in the training top-view image with horizontal frames to obtain a third top-view image; the training top-view image includes at least one person; Step 2: obtaining a fourth top-view image with an angle after being marked with a human body frame based on the third top-view image; Step 3: inputting the fourth top-view image into the rotating frame prediction network, so that the rotating frame prediction network outputs a training pre-selection frame; Step 4: Process the training pre-selected frame to obtain an angled training rotation frame corresponding to the human body; Step 5: Calculate the difference between the angle in the training rotation frame and the angle in the fourth top view image to obtain an angle loss value, and calculate the total loss value based on the angle loss value; Step 6: looping through steps 1 to 5 until the total loss value stops decreasing, and obtaining the trained rotation frame prediction network; The processing module is used to process the target pre-selected frame and obtain an angled target rotation frame corresponding to the pedestrian's head and shoulders.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for acquiring the head-shoulder angle of the top view through unsupervised learning according to any one of claims 1 to 5 is implemented.

8. A terminal, characterized in that: include: processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory, so that the terminal executes the unsupervised learning method for acquiring head-and-shoulder angles from a top perspective according to any one of claims 1 to 5.

Images