A system and method for detecting distance between multiple individuals based on group recognition

Abstract

The present invention discloses a method for detecting the distance between multiple individuals based on group recognition, comprising the following steps: step S1, using a preset deep neural network model and a group recognition method to identify all individuals in the picture, their information, and the groups they are in; step S2, using the obtained information such as the category, position, size, etc. of each individual in each group, and estimating the real position information and mutual distance of these individuals through an algorithm based on spatial modeling or an algorithm based on machine learning; step S3, using the obtained positional relationship between each individual in each group and all other individuals, identifying subgroups and classifying multiple inter-individual relationships through a classification algorithm, so as to realize the monitoring and control of these two or more inter-individual relationships in the group. The present invention proposes a real-time group supervision method that can be applied to different scenarios, and can be used for monitoring and management of social distance, social relations, and group behavior of two or more parties.

Description

Technical Field

[0001] The invention belongs to the field of distance detection of individuals of the same or different species in a group, and in particular relates to a system and method for detecting distance between multiple individuals based on group recognition. Background Art

[0002] In recent years, as my country's economy has gradually entered a stage of high-quality development from a stage of high-speed growth, group-level intelligent management is gradually replacing individual-level automated management, becoming an inevitable trend of the times. In order to monitor the group behavior of people or objects in real time, ensure group order, and achieve distance control of multiple individuals in response to sudden outbreaks of infectious diseases in humans or animals, the use of intelligent surveillance cameras to achieve unattended real-time automated monitoring of the relationship between individuals of the same or different species in a group has become an urgent need for intelligent group management.

[0003] The existing methods for monitoring groups mainly monitor, re-identify, and track individuals. The only current method for monitoring groups is behavior detection, which only gives the categories of normal or abnormal behavior of individuals or the entire group, or describes the movement of the group in a macro sense, and fails to quantitatively monitor the relationship between multiple individuals in the group. It is inefficient, low in accuracy, and only applicable to groups of people. At the same time, the existing methods for monitoring the distance between two individuals and estimating group density mainly use simple estimation methods with low accuracy, and do not take into account the impact of the spatial position structure of multiple parties on the relationship between multiple individuals, and do not combine deep learning technology and spatial algorithms of artificial intelligence at the same time.

[0004] In the supervision of group order, the behavior between multiple individuals in a group is often more important than the behavior of a single individual. For example, during the prevention and control of the new coronavirus, it is very important to effectively monitor and manage the social distance between multiple individuals in a crowd. In ball games, multi-individual distance detection can be used to study the optimal strategy of the team formation in the game based on the group evolution of the opposing sides. In smart ranches and breeding, the relationship identification of multiple individuals in a group of the same or different species of livestock is also very meaningful for smart breeding. In smart cities, monitoring the distance between people and vehicles, vehicles and vehicles, and other individuals of the same or different species in groups such as traffic groups can promote the development of smart transportation and smart life. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and to provide a system and method for detecting the distance between multiple individuals based on group recognition.

[0006] The present invention aims at the mutual relationship between multiple individuals of the same or different species in a group that needs to be supervised, makes up for the deficiency of the prior art that can only track and re-identify a single individual or detect its abnormal behavior, overcomes the problem of low accuracy and low efficiency in detecting abnormal behavior of a group, realizes an algorithm for detecting the distance between multiple individuals of the same or different species in a group, and thus proposes a real-time group supervision method that can be applied to different scenarios, and can be used for monitoring and management of social distance, social relationship and group behavior of two or more parties.

[0007] The present invention is achieved through the following technical solutions, a system and method for detecting the distance between multiple bodies based on group recognition, comprising the following steps:

[0008] (1) Using surveillance cameras and other equipment to obtain real-time dynamic images (Observation) of the group to be detected, and then using deep neural network models such as YOLO, Mask RCNN, Faster RCNN, Fast RCNN, FPN, FCN, VGG or SSD to identify all individuals in the image and their categories, locations, sizes and other information, and then use group recognition methods to identify all groups composed of these individuals.

[0009] (2) Based on the information of all individuals detected in step 1, an algorithm based on spatial modeling (including an algorithm based on individual height and an algorithm based on a horizontal plane) or an algorithm based on machine learning is used to estimate the true location information of these individuals, thereby estimating the true distance between each individual in each group and all other individuals.

[0010] (3) According to the actual distance between each individual and all other individuals in each group obtained in step 2, multiple inter-individual relationships in the group, including bilateral social distance and multilateral social distance, are counted, sub-groups are identified, and finally, a classification algorithm is used to identify and classify all multiple inter-individual distance relationships, thereby realizing the monitoring and control of these bilateral or multilateral inter-individual relationships in the group.

[0011] The method for group identification in step 1 includes extracting features of each frame from the video, and identifying all groups in the picture using clustering, support vector machine (SVM) or neural network methods.

[0012] The algorithm based on spatial modeling in step 2 assumes that the vertical viewing angle of the camera is α, the horizontal viewing angle is β, the angle between the plane where the lens is located and the plumb plane when the camera looks down is θ, and the height between the camera and the ground is H c The video screen does not need to maintain the aspect ratio. The width and height after arbitrary scaling are W p , H p, then the video image can be restored to a plane parallel to the plane where the camera lens is located with a height of H p , length is The distance from the plane of this imaginary imaging picture to the camera can be calculated as:

[0013]

[0014] When calculating the real position of an individual based on the spatial modeling algorithm in step 2, a rectangular coordinate system Γ is established on a horizontal plane parallel to the ground. Ground The coordinate system takes the projection of the camera on the horizontal plane as the origin, and the projection of the direction perpendicular to the plane where the lens is located on the horizontal plane is the positive direction of the x-axis, and the projection of the direction from the upper left corner to the upper right corner of the imaging screen on the horizontal plane is the positive direction of the y-axis.

[0015] In step 2, the algorithm based on the individual height is used to calculate the individual's true position in Γ Ground The specific method of calculating the x coordinate in the coordinate system is as follows: the category of the i-th individual in the group predicted by the neural network model in step 1 is used to estimate the true height H of the individual i , and predict the height of the individual in the picture as h i , then the x coordinate X of the true position of the individual can be solved according to the properties of similar triangles i for:

[0016]

[0017] In step 2, the real position of the individual is calculated using an algorithm based on the horizontal plane. Ground The specific method of the x coordinate in the coordinate system is: Assuming that the actual height of the individual or the height of the individual in the picture cannot be accurately predicted by the neural network model in step 1, then it is only necessary to record the coordinates of the bottom center of the identified individual in the picture (x i ,y i ), the coordinate system where the coordinate is located is Γ Picture The origin is the upper left corner of the screen, and the positive directions of the x-axis and y-axis are respectively the direction from the upper left corner to the upper right corner and the direction from the upper left corner to the lower left corner of the screen. Then, according to the properties of similar triangles, the true position of the individual can be deduced to be in Γ Ground The x-coordinate X in the coordinate system i for:

[0018]

[0019] The two algorithms based on spatial modeling in step 2 finally need to be based on the X iCalculate the y coordinate of the individual's true position by: i ,y i ) can be calculated that its coordinates on the imaginary imaging plane are For ease of calculation, remember the horizontal scaling factor At the same time, according to the properties of similar triangles, it can be deduced that the imaging position of the bottom center of the individual in the picture on the imaginary imaging plane is Γ Ground The x coordinate in the coordinate system is:

[0020]

[0021] Thus, the true position of the individual can be calculated in Γ Ground The y coordinate in the coordinate system is:

[0022]

[0023] The true position of each individual in the group is obtained in Γ Ground Coordinates in the coordinate system (X i ,Y i ), so that the actual distance D between any two individuals in the group on the horizontal plane can be calculated G for:

[0024]

[0025] Among them, (X1, Y1) and (X2, Y2) are the real positions of the two individuals calculated by the above method in Γ Ground Coordinates in the coordinate system.

[0026] The machine learning-based algorithm in step 2 includes using models such as support vector machine (SVM), decision tree (DecisionTree), AdaBoost algorithm or neural network, and using statistical learning methods to learn the mapping relationship between the true distance between all individuals in the group and the information obtained in step 1, thereby realizing the prediction and estimation of the true distance between each individual in the group and all other individuals.

[0027] The multi-party social distance in step 3 includes a. the overall social distance within the subgroups composed of relatively strong mutual relationships and b. the social distance between the subgroups composed of relatively strong mutual relationships.

[0028] In step 3, the relationships between multiple individuals are dynamically classified according to the location distribution and social distance of all individuals using threshold partitioning, graph theory or machine learning (including support vector machine (SVM), decision tree, AdaBoost algorithm or neural network, etc.).

[0029] In step 3, the changes in the relationships between multiple entities in the time domain use a finite state machine (FSM), a recurrent neural network (RNN), a long short-term memory network (LSTM) or its variant (GRU) to realize the state memory function.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) For groups, the present invention introduces the technology of group identification and multi-individual relationship detection, identifies and classifies the multi-individual relationships in the group, and thus analyzes the characteristics of the entire group. It can be used for the supervision and analysis of group behavior, including maintaining daily group order, supervising traffic order, responding to sudden outbreaks of infectious diseases in humans or animals, and analyzing the relationships within the group (including bilateral social distance and multi-party social distance).

[0032] (2) For the detection of relationships between multiple individuals, the present invention combines machine learning, deep learning and spatial geometry-related algorithms. By simply using existing ordinary surveillance cameras, the real position of each individual in the group can be accurately and in real time calculated.

[0033] (3) Aiming at the study of relationships between multiple individuals in a group, the present invention is based on professional methods such as social networks, combines traditional methods including graph theory with cutting-edge technologies such as graph neural networks and recursive neural networks, considers the changes in relationships between multiple individuals in the time domain, and establishes an adaptive dynamic group relationship analysis model to achieve the purpose of reducing the false detection rate.

[0034] (4) The present invention is applicable to groups composed of individuals of the same or different species. The group may include individuals of different types, including humans, animals, vehicles, and any other objects. It can be applied to various application scenarios such as social distance monitoring during sudden outbreaks of infectious diseases, team formation strategy research in multi-player competitions, smart farming, smart city smart transportation, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the overall process of the embodiment of the patent of the present invention;

[0036] Figure 2 This is a visualization effect diagram of the process of social distance monitoring of a pedestrian group in an embodiment of the present invention, wherein: (a) is a monitoring video, (b) is a schematic diagram of an identified individual in the picture, and (c) is a visualization effect diagram of the final social distance estimation based on the pedestrian group;

[0037] Figure 3 Schematic diagram of calculating the x-coordinate of an individual on a horizontal plane using an algorithm based on the individual's height.

[0038] Figure 4 Schematic diagram of calculating the x-coordinate of an individual on a horizontal plane using a horizontal plane-based algorithm.

[0039] Figure 5 Schematic diagram of calculating the y-coordinate of an individual on the horizontal plane using two algorithms based on spatial modeling. DETAILED DESCRIPTION

[0040] In order to more clearly illustrate the specific implementation method of the present invention and show the features and advantages of the present invention, the non-limiting embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0041] It should be noted that this embodiment only uses pedestrians as the only individuals in the group to illustrate the basic ideas, processes and technical principles of the present invention in a schematic manner. Its actual implementation may include many different types of individuals, including humans, animals, vehicles, and any other objects.

[0042] Participate in the Figure 1 , Figure 1 The main steps of the system and method for detecting the distance between multiple bodies based on group recognition according to an embodiment of the present invention are illustrated by way of example. Figure 1 As shown, the system and method for detecting multiple body distances based on group recognition in this embodiment may include step S1, step S2 and step S3.

[0043] Step S1, using a preset deep neural network model and a group recognition method to identify all individuals in the picture, their information, and the groups they belong to.

[0044] Further, step S1 may include S11, S12 and S13.

[0045] Step S11, obtain dynamic real-time images in real time through existing monitoring cameras and other equipment, as shown in the attached Figure 2 As shown in (a).

[0046] Step S12, using a preset deep neural network model to identify all individuals in the picture and their categories, positions, sizes and other information, as shown in the attached figure. Figure 2 The one shown in (b) is one of them.

[0047] Step S13, identifying all groups consisting of these individuals through a group identification method.

[0048] Furthermore, in step S11, in order to improve efficiency and meet real-time requirements, a frame may be extracted from the real-time video at a certain frequency f for analysis.

[0049] Furthermore, in step S11, the present invention does not impose any requirements on the size ratio of the video screen, and the size can be scaled arbitrarily without a fixed size. Therefore, in order to improve storage and processing efficiency, the video screen size is compressed to an appropriate size.

[0050] Step S2, using the obtained information such as the category, location, size, etc. of each individual in each group, estimates the real location information and mutual distance of these individuals through an algorithm based on spatial modeling (including an algorithm based on individual height and an algorithm based on horizontal plane) or an algorithm based on machine learning.

[0051] Further, step S2 may include S21 and S22.

[0052] Step S21, estimate the coordinates of the real position of each individual in each group through an algorithm based on spatial modeling (including an algorithm based on individual height and an algorithm based on a horizontal plane) or an algorithm based on machine learning. The calculation method can be selected according to the actual situation.

[0053] Specifically, in step S21, the algorithm based on spatial modeling assumes that the vertical viewing angle of the camera is α, the horizontal viewing angle is β, the angle between the plane where the lens is located and the plumb plane when the camera looks down is θ, and the height of the camera from the ground is H. c The video screen does not need to maintain the aspect ratio. The width and height after arbitrary scaling are W p , H p .

[0054] Specifically, in step S21, the method for calculating the x coordinate of the individual on the horizontal plane using an algorithm based on the individual height is as follows: Figure 3 As shown, o is the camera position, π is the imaginary imaging screen in front of the camera lens, and the height of the individual in the screen h i As attached Figure 2 As shown in (b), the individual category predicted by the neural network and the individual height inferred is H i , assuming that in one application of this embodiment, α=120°, θ=30°, H c =6(m),H p =0.1(m),H i =1.7(m),h i = 0.004 (m), then the distance from the plane of the imaginary imaging screen to the camera can be calculated as:

[0055]

[0056] Then the x-coordinate of the individual's true position is X i for:

[0057]

[0058] Specifically, in step S21, the method for calculating the x coordinate of an individual on the horizontal plane using an algorithm based on the horizontal plane is as follows: Figure 4 As shown, where o is the camera position, π is the imaginary imaging screen in front of the camera lens, P is the bottom center of the individual in the screen, and its coordinate system Γ in the screen Picture The coordinates on (x i ,y i ) as attached Figure 2 As shown in (b), assuming that in one application of this embodiment, α=120°, θ=30°, H c =6(m),H p =0.1(m),y i =0.06(m), by comparison, we can see that in this application 2y i >H p , then the distance from the plane of the imaginary imaging screen to the camera can be calculated as:

[0059]

[0060] So Figure 4 The x-coordinate X of the actual position of the individual i for:

[0061]

[0062] Specifically, in step S21, both algorithms based on spatial modeling need to be based on the X i Calculate the y coordinate, the specific method is as follows Figure 5 As shown, the xoy rectangular coordinate system is the rectangular coordinate system Γ established on the horizontal plane parallel to the real ground. Ground , o is the projection of the camera position on the ground, π is the imaginary imaging screen in front of the camera lens, P is the bottom center of the individual in the screen, and its coordinate system Γ in the screen Picture The coordinates on (x i ,y i ) as attached Figure 2 As shown in (b), assuming that in one application of this embodiment, α=120°, β=120°, θ=30°, H c =6(m),H p =0.1(m),W p =0.15(m),x i =0.03(m),y i = 0.06(m), then the lateral scaling factor can be calculated The x-coordinate X of the individual on the horizontal plane calculated using an algorithm based on the individual's height or an algorithm based on the horizontal plane i≈5.2(m), by comparison, we can see that in this application 2y i >H p , so that the individual's true position can be calculated in Γ Ground The y coordinate in the coordinate system is:

[0063]

[0064] Thus, the real position of the individual in this application is obtained in Γ Ground The coordinates in the coordinate system are (X i ,Y i )≈(5.2(m),7.8(m)).

[0065] Specifically, when using a machine learning-based algorithm to calculate the real position of an individual in step S21, it is necessary to first measure the actual distance multiple times in the real environment, take photos, and mark the actual coordinates of the individual's real position on them, make a data set and divide it into a training set, a validation set and a test set in the ratio of n:m:l, where n>m, n>l, and then train the specified machine learning model to predict the coordinates of the individual's real position, define a loss function to calculate the gap between the coordinates predicted by the model and the actual coordinates, and use an optimization algorithm to make the loss function value of the model prediction result converge on the training set and reach the minimum on the validation set, and the effect of this machine learning model can be tested on the test set.

[0066] Step S22, the method for estimating the real distance between the individual and all other individuals is: according to the real position coordinates of the two individuals obtained in step S21, which are (X1, Y1) and (X2, Y2), the distance D between the real positions of the two individuals on the horizontal plane can be calculated. G for:

[0067]

[0068] Step S3, using the positional relationship between each individual in each group and all other individuals, identify subgroups and classify multiple inter-individual relationships through a classification algorithm to monitor and control these two or more inter-individual relationships in the group, as shown in the attached figure. Figure 2 As shown in (c), this embodiment is an application of social distance management during the prevention and control of the new coronavirus epidemic. By detecting the multi-party social distance of each person, a low-risk threshold and a high-risk threshold are set to issue a warning to subgroups with risks in multi-party social distance.

[0069] So far, the technical solutions of the present invention have been described in conjunction with the implementation methods shown in the accompanying drawings, and the protection scope of the present invention is not limited to these specific implementations. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A method for detecting distance between multiple individuals based on group recognition, characterized in that: The steps include: Step S1 obtains a dynamic real-time picture of the group to be detected in real time through a surveillance camera, and then uses a deep neural network model including YOLO, Mask RCNN, Faster RCNN, Fast RCNN, FPN, FCN, VGG or SSD to identify all individuals in the dynamic real-time picture and their categories, locations, and size information, and then identifies all groups composed of these individuals through a group recognition method; Step S2 estimates the real position information of all individuals detected in step S1 using an algorithm based on spatial modeling, thereby estimating the real distance between each individual in each group and all other individuals, wherein the algorithm based on spatial modeling includes an algorithm based on individual height or an algorithm based on a horizontal plane; Step S3: based on the real distance between each individual and all other individuals in each group obtained in step S2, multiple inter-individual relationships in the group, including bilateral social distance and multi-party social distance, are counted, sub-groups are identified, and finally, all multiple inter-individual distance relationships are identified and classified using a classification algorithm, thereby realizing monitoring and control of these bilateral or multi-party multiple inter-individual relationships in the group; When the algorithm based on spatial modeling in step S2 calculates the real position of the individual, the video image is restored to a plane parallel to the plane where the camera lens is located with a height of H p , length is The distance to the camera is The imaginary imaging picture, where α is the vertical viewing angle of the camera and β is the horizontal viewing angle; The real position information described in step S2 is referenced to a rectangular coordinate system Γ established on a horizontal plane parallel to the ground. Ground , the coordinate system takes the projection of the camera on the horizontal plane as the origin, the projection of the direction perpendicular to the plane where the lens is located on the horizontal plane is the positive direction of the x-axis, and the projection of the direction from the upper left corner to the upper right corner of the imaging screen on the horizontal plane is the positive direction of the y-axis; The algorithm based on individual height described in step S2 estimates the true height H of the individual by using the category of the individual in the group predicted by the neural network model. i , and predict the height of the individual in the imaging screen as h i According to the properties of similar triangles, the true position of the individual is solved in Γ Ground The x-coordinate X in the coordinate system i for: Where θ is the angle between the plane where the lens is located and the plumb plane when the camera is looking down, H c is the height of the camera from the ground; the horizontal plane-based algorithm uses the coordinates of the bottom center of the individual in the video image identified by the neural network (x i ,y i ), according to the properties of similar triangles, the true position of the individual is deduced to be in Γ Ground The x-coordinate X in the coordinate system i for: The true position of the individual is in Γ Ground The y coordinate in the coordinate system can be obtained by using the actual position of the individual in Γ according to the properties of similar triangles. Ground The x-coordinate X in the coordinate system i It is deduced that the true position of the individual is in Γ Ground The y coordinate in the coordinate system is: The video screen does not need to maintain the aspect ratio. The width and height after arbitrary scaling are W p , H p , horizontal scaling factor 2. The method for detecting distances between multiple individuals based on group recognition according to claim 1, characterized in that: The individuals include humans, animals, and vehicles; the relationships between the multiple individuals include relationships between individuals of the same species or different species.

3. The method for detecting distances between multiple individuals based on group recognition according to claim 2, characterized in that: Relationships between individuals of the same species include social distance between humans and between animals.

4. The method for detecting distances between multiple individuals based on group recognition according to claim 1, characterized in that: The group recognition method in step S1 includes extracting features of each frame from the video, and using clustering, support vector machine or neural network methods to recognize all groups in the dynamic real-time picture.

5. The method for detecting distances between multiple bodies based on group recognition according to claim 1, characterized in that: Using the true position of each individual in the group in Γ Ground Coordinates in the coordinate system (X i ,Y i ), calculate the actual distance D between any two individuals in the group on the horizontal plane G for:

6. The method for detecting distance between multiple bodies based on group recognition according to claim 1, characterized in that: The multi-party social distance includes: a. the overall social distance within subgroups composed of strong mutual relationships; b. the social distance between subgroups composed of strong mutual relationships; the multi-individual relationships are dynamically classified according to the location distribution and social distance of all individuals using threshold division, graph theory or machine learning methods; the changes in the multi-individual relationships in the time domain use finite state machines, recursive neural networks, and long short-term memory networks to realize the state memory function.

Images