A graph relationship network population counting method and related equipment

Abstract

The present invention discloses a method for counting people in a graph relational network and related equipment, the method comprising: predicting a picture sequence to obtain a bounding box and a bounding box image; constructing a first relational subgraph set according to the bounding box; calculating graph similarity according to the correlation between adjacent nodes and the correlation between adjacent edges; calculating the average histogram similarity between the first bounding box image and the second bounding box image; calculating the comprehensive similarity of the graph similarity and the average histogram similarity, if the comprehensive similarity is greater than or equal to a preset threshold, replacing the second relational subgraph with the first relational subgraph, otherwise directly adding the first relational subgraph to the second relational subgraph set, so as to obtain the number of different detection objects in the picture sequence. By comparing the calculated comprehensive similarity with the preset threshold, and updating the number of different detection objects in the second relational subgraph set according to the comparison result, the number of different detection objects in the image is efficiently and accurately counted.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for counting people in a graph relationship network and related equipment. Background Art

[0002] With the development of society, the process of urbanization is also accelerating, and construction sites can be seen everywhere in many cities. However, construction sites have many facilities and complex personnel, which may pose many safety hazards. Personnel inventory at large construction sites is an important part of linear engineering and is of great significance to on-site resource allocation and personnel scheduling. Population statistics based on video surveillance can not only provide important reference data for public safety managers to formulate protection strategies, but also use these data to achieve effective crowd monitoring and safety guidance.

[0003] In traditional methods, the personnel technology on the construction site mostly adopts the method of manual on-site counting or uses a punch card machine to count the number of people on site. The former increases the consumption of manpower and is prone to omissions, and the latter method is not suitable for all construction sites because the turnover of personnel on the construction site is very large. With the emergence of various sensor equipment (radar and magnetometer, etc.), some researchers have also used these devices to count the number of people, but these devices have high installation requirements and are not convenient to install on a large area.

[0004] With the rapid development of drone technology and the continuous improvement of artificial intelligence technology, the combination of drones and computer theory has been widely used in the industrial field. Due to the flexibility and portability of drones, it is very convenient to use drones for population counting. At present, the people counting methods based on video surveillance are generally divided into two branches, namely, the people counting methods based on tracking and the people counting methods based on target detection. Among them, the people counting methods based on tracking use continuous video images or object trajectory information for people counting, which can effectively solve the problem of crowd occlusion, but if a person does not appear in the field of view of the drone for a long time, he will be regarded as a new person. The method based on target detection can obtain the number and location information of people at the same time, but in the case of high occlusion, high crowd density and large scale changes of the objects to be detected, the detection error rate and counting error are very high. Moreover, the method based on target detection also has the same problem. When the same person appears repeatedly in the field of view of the drone, it will be mistaken for a new person, resulting in counting errors.

[0005] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0006] The main purpose of the present invention is to provide a graph relational network people counting method and related equipment, aiming to solve the problem of low accuracy of people counting methods based on video surveillance in the prior art.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for counting people in a graph relationship network, the method comprising the following steps:

[0009] Using the detection network to predict the input image sequence, obtain the key point positions, bounding boxes and categories of all detection objects in the image sequence, and the bounding box sizes corresponding to the key points, and obtain the bounding box images of the image sequence according to the key point positions, the categories and the bounding box sizes;

[0010] Constructing a corresponding first relationship subgraph according to the bounding box of the detected object, after obtaining a first relationship subgraph set of all detected objects, initializing a second relationship subgraph set and querying the first relationship subgraph, and directly adding the first relationship subgraph that appears for the first time to the second relationship subgraph set;

[0011] Normalizing and iteratively calculating the correlations between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlations between adjacent edges in the first relationship subgraph and the second relationship subgraph, respectively, to obtain graph similarity between the first relationship subgraph and the second relationship subgraph;

[0012] Calculating the histogram similarity of the first bounding box image and the second bounding box image on three channels respectively, and calculating the average histogram similarity of the first bounding box image and the second bounding box image;

[0013] Calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity;

[0014] Calculate the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs respectively, and compare them with a preset threshold value; if the comprehensive similarity is not less than the preset threshold value, replace the second relationship subgraph with the first relationship subgraph; otherwise, directly add the first relationship subgraph to the second relationship subgraph set to update the second relationship subgraph set;

[0015] When all the first relationship subgraphs are calculated, the number of the second relationship subgraphs in the updated second relationship subgraph set is calculated to obtain the number of different detection objects in the picture sequence.

[0016] In the graph relational network people counting method, the step of using the detection network to predict the input picture sequence, obtaining the key point positions, bounding boxes and categories of all detection objects in the picture sequence, and the bounding box sizes corresponding to the key points, and obtaining the bounding box images of the picture sequence according to the key point positions, the categories and the bounding box sizes specifically includes:

[0017] Compressing the picture sequence in units of step length to obtain a compressed image, using the detection network to predict the key point positions and categories of all the detection objects in the compressed image, and simultaneously obtaining a bounding box;

[0018] Predicting a corresponding key point heat map according to each of the categories, and splicing all the key point heat maps to obtain a predicted heat map, and then calculating a loss value according to the predicted heat map and a reference heat map of the category to obtain the bounding box size;

[0019] The bounding box image of the picture sequence is restored according to the key point position, the category and the bounding box size of the compressed image.

[0020] In the graph relation network population counting method, the steps of constructing a corresponding first relation subgraph according to the bounding box of the detected object, obtaining a first relation subgraph set of all detected objects, initializing a second relation subgraph set and querying the first relation subgraph, and directly adding the first relation subgraph that appears for the first time to the second relation subgraph set specifically include:

[0021] A relationship graph is constructed by taking the bounding box of the detected object as a central node and other adjacent detected objects as neighbor nodes;

[0022] Refining the relationship graph, and constructing a relationship subgraph according to the refined relationship graph to obtain the first relationship subgraph set of all the detection objects;

[0023] After initializing an empty second relationship subgraph set, query the first relationship subgraph in the first relationship subgraph set, and directly add the first relationship subgraph that appears for the first time to the second relationship subgraph set.

[0024] In the method for counting people in a graph relationship network, the step of performing normalization processing and iterative calculation on the correlations between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlations between adjacent edges in the first relationship subgraph and the second relationship subgraph, respectively, to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph specifically includes:

[0025] After respectively calculating the correlation between adjacent nodes in the first relationship subgraph and the correlation between adjacent nodes in the second relationship subgraph, normalization processing is performed to obtain the node similarity between the first relationship subgraph and the second relationship subgraph, wherein the second relationship subgraph is the first relationship subgraph added to the second relationship subgraph set;

[0026] After respectively calculating the correlation between adjacent edges in the first relationship subgraph and the correlation between adjacent edges in the second relationship subgraph, normalization processing is performed to obtain edge similarities between adjacent edges in the first relationship subgraph and adjacent edges in the second relationship subgraph;

[0027] Iteratively calculating the node similarity and the edge similarity to obtain the node similarity matrix and the edge similarity matrix;

[0028] The corresponding positions in the node similarity matrix are compared to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph.

[0029] In the graph relation network people counting method, the steps of respectively calculating the histogram similarities of the first bounding box image and the second bounding box image on three channels and calculating the average histogram similarity of the first bounding box image and the second bounding box image include:

[0030] Adjusting the first bounding box image corresponding to the first relationship subgraph and the second bounding box image corresponding to the second relationship subgraph to a preset size, wherein the first bounding box image is the bounding box image corresponding to the first relationship subgraph, and the second bounding box image is the bounding box image corresponding to the second relationship subgraph;

[0031] Separating the adjusted first bounding box image and the adjusted second bounding box image in three channels respectively, and calculating corresponding histogram similarities in the three channels respectively;

[0032] An average value of the histogram similarities of the three channels is calculated to obtain an average histogram similarity of the first bounding box image and the second bounding box image.

[0033] In the method for counting people in a graph relationship network, the step of calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph based on the graph similarity and the average histogram similarity specifically includes:

[0034] Substituting a preset ratio into the graph similarity and the average histogram similarity for calculation, a comprehensive similarity between the first relationship subgraph and the second relationship subgraph is obtained.

[0035] In the graph relationship network population counting method, the steps of respectively calculating the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs and comparing the similarity with a preset threshold, and replacing the second relationship subgraph with the first relationship subgraph if the comprehensive similarity is not less than the preset threshold, otherwise, directly adding the first relationship subgraph to the second relationship subgraph set to update the second relationship subgraph set specifically include:

[0036] respectively calculating the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs, and comparing the similarity with the preset threshold each time;

[0037] If the comprehensive similarity is greater than or equal to the preset threshold, the first relationship subgraph replaces the second relationship subgraph; if the comprehensive similarity is less than the preset threshold, the first relationship subgraph is directly added to the second relationship subgraph set to update the second relationship subgraph set.

[0038] A graph relationship network population counting system, the graph relationship network population counting system comprising:

[0039] A bounding box image generation module is used to predict the input image sequence using the detection network, obtain the key point positions, bounding boxes and categories of all detection objects in the image sequence, and the bounding box sizes corresponding to the key points, and obtain the bounding box image of the image sequence according to the key point positions, the categories and the bounding box sizes;

[0040] A second relationship subgraph generation module is used to construct a corresponding first relationship subgraph according to the bounding box of the detected object, and after obtaining the first relationship subgraphs of all detected objects, initialize the second relationship subgraph and query the first relationship subgraph, and directly add the first relationship subgraph that appears for the first time to the second relationship subgraph;

[0041] A graph similarity calculation module, used to respectively normalize and iteratively calculate the correlation between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlation between adjacent edges in the first relationship subgraph and the second relationship subgraph, to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph;

[0042] an average histogram similarity calculation module, used to calculate the histogram similarity of the first bounding box image and the second bounding box image on three channels respectively, and calculate the average histogram similarity of the first bounding box image and the second bounding box image;

[0043] A comprehensive similarity solving module, used for calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity;

[0044] A preset threshold comparison module is used to calculate the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs respectively, and compare them with the preset threshold. If the comprehensive similarity is not less than the preset threshold, the second relationship subgraph is replaced by the first relationship subgraph; otherwise, the first relationship subgraph is directly added to the second relationship subgraph set to update the second relationship subgraph set;

[0045] The detection object quantity generation module is used to calculate the number of the second relation subgraphs in the updated second relation subgraph set after all the first relation subgraphs are calculated, so as to obtain the number of different detection objects in the picture sequence.

[0046] A terminal comprises: a memory, a processor, and a graph relationship network population counting program stored in the memory and executable on the processor, wherein the graph relationship network population counting program implements the steps of the graph relationship network population counting method as described above when executed by the processor.

[0047] A computer-readable storage medium stores a graph relationship network population counting program, and when the graph relationship network population counting program is executed by a processor, the steps of the graph relationship network population counting method described above are implemented.

[0048] Compared with the prior art, the present invention provides a graph relational network people counting method and related equipment, the method comprising the following steps: using a detection network to predict an input image sequence, obtaining key point positions, bounding boxes and categories of all detection objects in the image sequence, and the bounding box sizes corresponding to the key points, and obtaining the bounding box images of the image sequence according to the key point positions, the categories and the bounding box sizes; constructing a corresponding first relational subgraph according to the bounding boxes of the detection objects, and after obtaining the first relational subgraph set of all detection objects, initializing the second relational subgraph set and querying the first relational subgraph, and directly adding the first relational subgraph that appears for the first time to the second relational subgraph set; respectively normalizing and iteratively calculating the correlations between adjacent nodes in the first relational subgraph and the second relational subgraph, and the correlations between adjacent edges in the first relational subgraph and the second relational subgraph, to obtain The method comprises the steps of: calculating the graph similarity between the first relationship subgraph and the second relationship subgraph; calculating the histogram similarity of the first bounding box image and the second bounding box image on three channels, and calculating the average histogram similarity of the first bounding box image and the second bounding box image; calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity; calculating the comprehensive similarity between each of the first relationship subgraph and all the second relationship subgraphs, and comparing them with a preset threshold value; if the comprehensive similarity is not less than the preset threshold value, replacing the second relationship subgraph with the first relationship subgraph; otherwise, directly adding the first relationship subgraph to the second relationship subgraph set to update the second relationship subgraph set; when all the first relationship subgraphs are calculated, calculating the number of the second relationship subgraphs in the updated second relationship subgraph set to obtain the number of different detection objects in the image sequence. After calculating the graph similarity between the first relationship subgraph and the second relationship subgraph, and the average histogram similarity between the first bounding box image corresponding to the first relationship subgraph and the second bounding box image corresponding to the second relationship subgraph, the comprehensive similarity of the graph similarity and the average histogram similarity is solved and compared with a preset threshold, and the second relationship subgraph set is updated accordingly according to the comparison result to obtain the number of different detection objects in the image sequence, thereby efficiently and accurately counting the number of different detection objects in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a preferred embodiment of a method for counting people in a graph relationship network provided by the present invention;

[0050] Figure 2 A schematic diagram of a bounding box image in a preferred embodiment of the method for counting people in a graph relational network provided by the present invention;

[0051] Figure 3 A framework diagram of a preferred embodiment of the method for counting people in a graph relationship network provided by the present invention;

[0052] Figure 4 A flowchart of step S100 in a preferred embodiment of the method for counting people in a graph relationship network provided by the present invention;

[0053] Figure 5 A flowchart of step S200 in a preferred embodiment of the method for counting people in a graph relationship network provided by the present invention;

[0054] Figure 6 A schematic diagram of the construction process of the first relationship subgraph in a preferred embodiment of the graph relationship network population counting method provided by the present invention;

[0055] Figure 7 A flowchart of step S300 in a preferred embodiment of the method for counting people in a graph relationship network provided by the present invention;

[0056] Figure 8 A schematic diagram of a node relationship subgraph in a preferred embodiment of the method for counting people in a graph relationship network provided by the present invention;

[0057] Fig. 9 A schematic diagram of adjacent edge relationship subgraphs in a preferred embodiment of the graph relationship network population counting method provided by the present invention;

[0058] Fig.10 A flowchart of step S400 in a preferred embodiment of the method for counting people in a graph relationship network provided by the present invention;

[0059] Fig.11 A flowchart of step S600 in a preferred embodiment of the method for counting people in a graph relationship network provided by the present invention;

[0060] Fig.12 The principle block diagram of the graph relation network population counting system provided by the present invention;

[0061] Fig.13 A schematic diagram of an operating environment of a preferred embodiment of the terminal provided by the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0064] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0065] The present invention provides a method for counting people in a graph relational network and related equipment. In the present invention, after calculating the node similarity between the first relational subgraph and the second relational subgraph according to the correlation between adjacent nodes in the first relational subgraph and the second relational subgraph, iterative calculation is performed to obtain the graph similarity of the first relational subgraph and the second relational subgraph, and then the average histogram similarity of the first bounding box image corresponding to the first relational subgraph and the second bounding box image corresponding to the second relational subgraph is calculated, and after solving the comprehensive similarity of the graph similarity and the average histogram similarity, it is compared with a preset threshold, and the second relational subgraph set is updated accordingly according to the comparison result, so as to obtain the number of different detection objects in the image sequence, thereby efficiently and accurately counting the number of different detection objects in the image.

[0066] The following describes the design of the method for counting people in a graph relationship network through specific exemplary embodiments. It should be noted that the following embodiments are only used to explain the technical solution of the invention and are not specifically limited:

[0067] See also Figure 1 The present invention provides a method for counting people in a graph relationship network, and the method for counting people in a graph relationship network comprises the following steps:

[0068] S100, using the detection network to predict the input image sequence, obtain the key point positions, bounding boxes and categories of all detection objects in the image sequence, and the bounding box sizes corresponding to the key points, and obtain the bounding box image of the image sequence according to the key point positions, the categories and the bounding box sizes;

[0069] Specifically, firstly, the video shot by the drone is segmented into the image sequence: Among them, W is the image width, H is the image height, k is the number of pictures, and each picture can be represented by a W×H×3 matrix Represented; then input into the detection network for prediction to detect all the detection objects in each picture. In this embodiment, people are taken as an example, and a bounding box corresponding to each detection object (the bounding box contains the key point position and category information of the detection object) and the size of the bounding box corresponding to the key point are obtained: Where r is the step size, and then the bounding box image of the picture sequence is obtained according to the key point position, the category and the bounding box size: k is the number of bounding box images, as follows Figure 2 As shown; wherein the detection network is based on the CenterNet network, which can directly predict the center coordinates (key points), length, width and category c of the bounding box, for example, see Figure 3 In the left part of , for N input images, the detected objects are obtained through the CenterNet network, including the center coordinates (key point position), the length and width of the detection box (bounding box) and the category of the detection object, so as to obtain the corresponding bounding box image and relationship subgraph.

[0070] For further information, see Figure 4 The step of using the detection network to predict the input image sequence, obtaining the key point positions, bounding boxes and categories of all detection objects in the image sequence, and the bounding box sizes corresponding to the key points, and obtaining the bounding box image of the image sequence according to the key point positions, the categories and the bounding box sizes specifically includes:

[0071] S110, compressing the picture sequence in units of step length to obtain a compressed image, using the detection network to predict the key point positions and categories of all the detection objects in the compressed image, and simultaneously obtaining a bounding box;

[0072] S120, predicting a corresponding key point heat map according to each of the categories, and splicing all the key point heat maps to obtain a predicted heat map, and then calculating a loss value according to the predicted heat map and a reference heat map of the category to obtain the bounding box size;

[0073] S130: Restore the bounding box image of the picture sequence according to the key point position, the category and the bounding box size of the compressed image.

[0074] Specifically, for the picture sequence: Each picture in is compressed with a step length r to obtain a compressed image Then, the compressed image is used as the input of the detection network, and a neural network is used to predict a key point heat map for each category c: (The heat map exists in the form of a matrix), that is, a prediction is made for each position in the compressed map, where It means that the position x, y in the compressed image is considered to be category c, and the bounding box is obtained at the same time. Then, the key point heat maps of c categories are spliced ​​to obtain the predicted heat map: Where W, H, C and r are width, height, number of key point categories and step length respectively, for example W = 256, H = 256, C = 2, r = 4;

[0075] And for each category c in the image, there is a baseline heatmap: (The reference heat map is used to compare with the predicted key point heat map); the reference heat map is generated by the Gaussian function of the reference heat map (the reference heat map has the maximum value, which is 1, only when it is on the key point), and its function expression is as follows:

[0076]

[0077] The meaning of formula (1) is that for each category in each image, a Then there are c categories and c Y x, , spliced ​​to get Y x,, , σ p is the adaptive standard deviation, which is set to 1 here. is a hyperparameter, a constant value, C is a category set, Y x,, It is expressed as the magnitude of the thermal value in category c with the horizontal axis being x and the vertical axis being y. The position coordinates of each category of objects in the image after compression.

[0078] For each category, such as people, if an image has multiple objects of the same category, multiple Gaussian functions overlap, and for the position (x, y) in the position compression map, the maximum value among multiple Gaussian functions is selected, and finally the reference heat map can be obtained. Y x,y,The meaning is the corresponding heat map value of the coordinate x, y category c, wherein in the position compression map, the closer the position is to the object of the corresponding category, the larger the value is.

[0079] Secondly, according to the prediction heat map (predicted key point position) and the baseline heat map Y obtained by formula (1) x,, (real key point position) calculates the predicted coordinate loss value, wherein the predicted coordinate loss value is a logistic regression function with reduced pixel penalty with focal loss, and uses the reference heat map to measure whether the predicted key point heat map is accurate. The more similar, the smaller the loss. The function expression of the predicted coordinate loss value is as follows:

[0080]

[0081] Among them, α and β are hyperparameters of the function, where α = 2, β = 4, and N is the number of key points in each image (i.e., the total number of all detected objects); represents the predicted key point position, Y xyc represents the real key point position. Based on the above formula (2), the key point position and category can be obtained, and When , it means that the (x, y) position of the compressed image is considered to be category c.

[0082] However, because the prediction is made on the compressed image, there may be a positional offset when the compressed image is restored to the original image. Therefore, in order to eliminate the positional deviation caused by compression, an offset is predicted for each key point: Here, only the original image position and the compressed image position are used for training, and the position deviation loss function is obtained as follows:

[0083]

[0084] in, is the coordinates of the key points in the compressed image (real coordinates), p is the position of the key points after the compressed image is expanded back to its original size (original image) (predicted coordinates), r is the compression step size, and N is the number of key points. Based on the above formulas (2) and (3), the position (center point) and category of the key points in the original image can be obtained. Here, the position obtained by prediction Combined position deviation Expand back to the original image with a step size of r.

[0085] Furthermore, all nodes whose heat value is greater than the surrounding 8 points are screened out (for each position of the compressed graph, there is a node, and a node greater than the surrounding 8 nodes means that it is larger than the nodes adjacent to it), and for each category, the top 100 peak nodes (the top 100 key points with the largest heat value) are retained for the training of the bounding box model. For category c, that is, The size of the peak node is Where c represents the cth category, for example, the number of categories is 2 (human and machine). And for each predicted key point, a bounding box size is predicted: The loss function for the bounding box size is as follows, which calculates the difference between the predicted bounding box size and the true bounding box size:

[0086]

[0087] Among them, p k represents the kth key point, represents the predicted bounding box size of the kth keypoint, S pk is the real bounding box size of the kth key point, and N is the number of key points, such as 100 here. For each bounding box size, it is expressed in the form of (width, height), such as S pk =(4,6) means that the true bounding box of the kth key point is 6 high and 4 wide.

[0088] Combining formulas (2), (3) and (4), we can obtain the overall loss function expression of the detection network:

[0089] L det =L k +λ o L o +λ s L s (5);

[0090] Wherein, formula (5) is a weighted combination of formula (2) (prediction coordinate loss), formula (3) (position offset loss) and formula (4) (prediction bounding box size loss) (the sum of coefficients does not have to be 1), λ o ,λ s Both are weight hyperparameters and can be set to 1 and 0.1 respectively.

[0091] The loss function for the bounding box size is combined with the predicted heatmap and the benchmark heatmap Y x,, The loss is used to measure whether the predicted position is accurate, the position deviation loss is used to generate a position deviation for each category, and the bounding box loss is used to measure the size of the predicted bounding box.

[0092] Finally, for each image sequence Use the key point positions, categories, and bounding box sizes in the compressed image to infer the key point positions, categories, and bounding box sizes of the original image, and obtain the corresponding detection results (identify multiple detection objects): And for each detection result (each object) There will be a corresponding bounding box to frame the picture The bounding box image is intercepted and saved to obtain the bounding box image:

[0093] For further information, please refer to Figure 1 S200: construct a corresponding first relationship subgraph according to the bounding box of the detected object, and after obtaining the first relationship subgraph set of all detected objects, initialize the second relationship subgraph set and query the first relationship subgraph, and directly add the first relationship subgraph that appears for the first time to the second relationship subgraph set;

[0094] Specifically, after the detection network predicts the bounding box of the detection object, a corresponding relationship graph is constructed according to the bounding box: The relationship graph can also be called a graph sequence or a directed graph, which is composed of nodes of the human category pointing to other nodes. The directed graph consists of a finite set of nodes. (where l is the number of nodes), an adjacency matrix Node feature matrix (where d is the extracted feature dimension) and the edge feature matrix ε i Composition; among them nodes is the key node identified in the detection network (i.e., the node containing the category, human or machine), and the edge feature matrix ε i Generated by the distance between nodes, node features Use node labels to represent this.

[0095] Then, the first relationship subgraph of all detected objects is constructed according to the relationship graph: Among them, a j ,f j ,e j Corresponding to A i , ε i The meaning is the same as that of Z, and the first relation subgraph set of all detected objects is obtained at the same time: Z i ,in, m is the number of people in the subgraph.

[0096] Next, in order to finally calculate the accurate number of different detection objects in the relationship subgraph set, after constructing the first relationship subgraph set of all detection objects according to the bounding box, an empty second relationship subgraph set Z is initialized, and then all first relationship subgraphs in the first relationship subgraph set are queried, and the first relationship subgraph that appears for the first time is directly added to the second relationship subgraph set as the initial second relationship subgraph in the second relationship subgraph set Z. The second relationship subgraph set is the global subgraph set Z, and Z0 is used as the initial global subgraph set.

[0097] For further information, see Figure 5 The step of constructing a corresponding first relationship subgraph according to the bounding box of the detected object, obtaining a first relationship subgraph set of all detected objects, initializing a second relationship subgraph set and querying the first relationship subgraph, and directly adding the first relationship subgraph that appears for the first time to the second relationship subgraph set specifically includes:

[0098] S210, constructing a relationship graph with the bounding box of the detection object as a central node and other adjacent detection objects as neighbor nodes;

[0099] S220, refining the relationship graph, and constructing a relationship subgraph according to the refined relationship graph to obtain the first relationship subgraph set of all the detection objects;

[0100] S230: After initializing an empty second relationship subgraph set, query the first relationship subgraph in the first relationship subgraph set, and directly add the first relationship subgraph that appears for the first time to the second relationship subgraph set.

[0101] Specifically, after the image sequence is input into the detection network to obtain the bounding box of the detection object, the relationship graph is constructed with the detection box as the central node and other adjacent detection objects as neighbor nodes: Then use the node-edge adjacency matrix A s and the endpoint node-edge adjacency matrix A t To refine the topological structure of the relationship graph, where A s and A t It is a matrix consisting of the number of nodes and the number of edges. i, j∈j0,1j indicates whether the node is the terminal of the edge. i and j represent the i-th row and j-th column respectively. i, j∈j0,1j means that each element of the matrix is ​​0 or 1. Indicates that node i is the starting point of edge j, Indicates that node i is the end point of edge j. Specifically, if node v i It is edge j The terminal (ie t(e j )=v i ),So If the node v i It is edge j The starting point (i.e. s(e j )=v i ),So And a special type of node-edge critical matrix A=A is given s AtT and in means s(e i )=t(ej ) That is, the starting point of edge i is the end point of edge j.

[0102] Then, a relationship subgraph is constructed based on the refined relationship graph: Where n is the number of relationship subgraphs, and the construction process of the relationship subgraph is as follows: Figure 6 , Figure 6 It represents that a relation subgraph of the detection target K is constructed according to the relation graph of the detection target K, so that the first relation subgraph can be obtained, and at the same time, the first relation subgraph set of all the detection objects can be obtained: Z i .

[0103] Next, in order to finally calculate the accurate number of different detection objects in the relationship subgraph set, after constructing the first relationship subgraph set of all detection objects according to the bounding box, an empty second relationship subgraph set Z is initialized, and then all first relationship subgraphs in the first relationship subgraph set are queried, and the first relationship subgraph that appears for the first time is directly added to the second relationship subgraph set as the initial second relationship subgraph in the second relationship subgraph set Z. The second relationship subgraph set is the global subgraph set Z, and Z0 is used as the initial global subgraph set.

[0104] For further information, please refer to Figure 1 , S300, respectively normalizing and iteratively calculating the correlations between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlations between adjacent edges in the first relationship subgraph and the second relationship subgraph, to obtain graph similarity between the first relationship subgraph and the second relationship subgraph;

[0105] Specifically, it is first necessary to calculate the correlation between adjacent nodes in the first relationship subgraph and the correlation between adjacent nodes in the second relationship subgraph, so as to calculate the node similarity between the first relationship subgraph and the second relationship subgraph, and then iteratively calculate the node similarity to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph.

[0106] For further information, see Figure 7 The step of performing normalization processing and iterative calculation on the correlations between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlations between adjacent edges in the first relationship subgraph and the second relationship subgraph respectively to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph specifically includes:

[0107] S310, respectively calculating the correlation between adjacent nodes in the first relationship subgraph and the correlation between adjacent nodes in the second relationship subgraph, and performing normalization processing to obtain the node similarity between the first relationship subgraph and the second relationship subgraph; wherein the second relationship subgraph is the first relationship subgraph added to the second relationship subgraph set;

[0108] S320, respectively calculating the correlation between adjacent edges in the first relationship subgraph and the correlation between adjacent edges in the second relationship subgraph, and performing normalization processing to obtain edge similarities between adjacent edges in the first relationship subgraph and adjacent edges in the second relationship subgraph;

[0109] S330, iteratively calculate the node similarity and the edge similarity to obtain the node similarity matrix and the edge similarity matrix

[0110] S340: Compare corresponding positions in the node similarity matrix to obtain graph similarity between the first relationship subgraph and the second relationship subgraph.

[0111] Specifically, see Figure 8 , taking a neighbor node u, i in the first relationship subgraph and a neighbor node v, j in the second relationship subgraph as an example, define the correlation r of the neighbor node u, i and the neighbor node v, j respectively ui and r vj as follows:

[0112]

[0113] Among them, d u ,d i ,d v and d j They are u node label, i node label, v node label and j node label, c ui is the distance between node u and node i and c vj is the distance between node v and node j. For target categories that are easy to move (such as people), lower label values ​​can be defined, and for target categories that are difficult to move (such as machines), higher label values ​​can be defined.

[0114] Then, the correlation between the adjacent nodes is obtained and normalized to obtain the node similarity between the adjacent nodes in the first relationship subgraph and the adjacent nodes in the second relationship subgraph, that is, the node similarity between the adjacent nodes in different directed graphs is obtained, and the formula is as follows:

[0115]

[0116] Among them, h ijRepresents the node similarity between node i in the first relationship subgraph and node j in the second relationship subgraph;

[0117] Similarly, the calculation of the edge similarity is the same as the calculation of the node similarity, please refer to Fig. 9 , for the adjacent edges e of neighbor nodes u and i in the first relationship subgraph u and e i , and the edge neighbors e of neighbor nodes v and j in the second relationship subgraph v and e j , define the correlation of edge neighbors as follows:

[0118]

[0119] Among them, d iu For the connecting edge e i and edge u The node type and d jv For the connecting edge e j and edge v Node category, c i and c u For edge e i and edge u Length.

[0120] Then, after the correlation of each adjacent edge is obtained, normalization processing is also performed to obtain the edge similarity between the adjacent edges in the first relationship subgraph and the adjacent edges in the second relationship subgraph, that is, the edge similarity between the adjacent edges in different directed graphs is obtained, and the formula is as follows:

[0121]

[0122] Next, the node similarity and the edge similarity are iteratively calculated to obtain a node similarity matrix M between the first relationship subgraph and the second relationship subgraph. N , and the edge similarity matrix M between the first relationship subgraph and the second relationship subgraph E , the specific formula is as follows:

[0123]

[0124] Among them, the above formula (10) is to calculate the node similarity matrix of the kth generation respectively and the edge similarity matrix of the kth generation When , the edge similarity matrix of the k-1th generation is needed and the node similarity matrix of the k-1th generation A is a directed graph A: The adjacency matrix of , B is a directed graph B: The adjacency matrix of is the initial node similarity matrix The corresponding element is h ij , is the initial marginal correlation matrix The corresponding element is h i ' j , k is the number of iterations, A s (or A t ) and B s (or B t ) are both node-edge adjacency matrices, A T is the transpose of the adjacency matrix of the directed graph A, (A s ) T is the transpose of the node-edge adjacency matrix.

[0125] Finally, compare the node similarity matrix between the first relationship subgraph and the second relationship subgraph The corresponding position of the same detection object in the first relationship subgraph and the second relationship subgraph are obtained. g , that is, comparing the node similarity matrix between two directed graphs The corresponding position of the same person in the image is obtained, and the graph similarity S between the directed graphs represented by the two characters is obtained. g ,like The number of rows is the number of nodes in A, and the number of columns is the number of nodes in B. Then the vth row and the uth column represent The node u and The graph similarity of node v.

[0126] For further information, please refer to Figure 1 , S400, calculating the histogram similarity of the first bounding box image and the second bounding box image on three channels respectively, and calculating the average histogram similarity of the first bounding box image and the second bounding box image;

[0127] Specifically, the histogram similarities of the first bounding box image corresponding to the first relationship subgraph and the second bounding box image corresponding to the second relationship subgraph on the three channels of R, G, and B are calculated respectively, so as to obtain the average histogram similarity between the first bounding box image and the second bounding box image.

[0128] For further information, see Fig.10 The step of calculating the histogram similarity of the first bounding box image and the second bounding box image on three channels respectively in S400, and calculating the average histogram similarity of the first bounding box image and the second bounding box image comprises:

[0129] S410: Adjust the first bounding box image corresponding to the first relationship subgraph and the second bounding box image corresponding to the second relationship subgraph to a preset size, wherein the first bounding box image is the bounding box image corresponding to the first relationship subgraph, and the second bounding box image is the bounding box image corresponding to the second relationship subgraph;

[0130] S420, separating the adjusted first bounding box image and the adjusted second bounding box image in three channels respectively, and calculating corresponding histogram similarities in the three channels respectively;

[0131] S430 , calculating an average value of the histogram similarities of the three channels to obtain an average histogram similarity of the first bounding box image and the second bounding box image.

[0132] Specifically, the first bounding box image corresponding to the first relationship subgraph The second bounding box image corresponding to the second relationship subgraph For example:

[0133] First, the first bounding box image and the second bounding box image After adjusting to the preset size, for example 256*256, for color images, it can be represented as a 256*256*3 matrix, and the corresponding separation is performed on the three channels of R, G, and B. And extract the adjusted first bounding box image respectively and the adjusted second bounding box image Pixel Channel Then, the adjusted first bounding box image is calculated on the three channels of R, G, and B respectively. and the adjusted second bounding box image The first bounding box image after adjustment on the three channels of R, G, and B is obtained accordingly and the adjusted second bounding box image Histogram similarity of: S r , S g and S b ; Finally, the average value of the histogram similarities of the three channels is calculated to obtain the average histogram similarity of the first bounding box image and the second bounding box image:

[0134] Among them, the algorithm of average histogram similarity is expressed as follows:

[0135] Algorithm 1:

[0136] Input: First bounding box image and the second bounding box image

[0137] Output: S h : and The average histogram similarity of

[0138] 1: Adjustment and The size is 256*256

[0139] 2: and Separated into three channels: R, G, and B

[0140] 3: Extract separately and Pixel Channel

[0141] 4:for p in range(r,g,b)do

[0142] 5: Initialization: d = 0 / / d is the histogram overlap

[0143] 6: for q = 0, q < 256 * 256 do / / P represents the pixel value

[0144] 7:if / / That is, determine the bounding box image In channel p, are the pixel values ​​at position q equal?

[0145] 8:

[0146] 9:else

[0147] 10:d=d+1

[0148] 11: p=p+1

[0149] 12: end for

[0150] 13:

[0151] 14: end for

[0152] 15:

[0153] 16:return S h .

[0154] The process of Algorithm 1 is described as follows:

[0155] For the first bounding box image of the input and the second bounding box image It can be decomposed into three matrices of 256*256*1 for three channels. And for each channel matrix, initialize d=0. For each position in the matrix, use (in, Represents a bounding box image In channel p, the pixel value at position q, Represents a bounding box image In channel p, the pixel value at position q) is used to measure the similarity between them, which is 1 if and only if the pixel values ​​at corresponding positions are equal. The average of 256*256 similarities It represents the similarity of the channel matrix, that is, d is the sum of the similarities of 256*256 positions. Finally, the average similarity of the three channel matrices is expressed as the average histogram similarity S h ,Right now

[0156] For further information, please refer to Figure 1 , S500, calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity;

[0157] Specifically, after calculating the average histogram similarity S between the first bounding box image and the second bounding box image, h Similarity S g After that, the comprehensive similarity between the first relationship subgraph and the second relationship subgraph can be further calculated.

[0158] Furthermore, S500, the step of calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity includes:

[0159] S510. Substitute a preset ratio into the graph similarity and the average histogram similarity to perform calculations to obtain a comprehensive similarity between the first relationship subgraph and the second relationship subgraph; wherein the preset ratio of the graph similarity and the preset ratio of the average histogram similarity add up to “1”.

[0160] Specifically, after obtaining the average histogram similarity S between the first bounding box image and the second bounding box image h Similarity S g Then, the comprehensive similarity between the character i in the first relationship sub-graph and the character j in the second relationship sub-graph can be obtained by calculating according to the preset ratio. For example, see Figure 3 On the right side, for the first relationship subgraph obtained, the average histogram similarity Sh Similarity S g , so as to remove the duplicated identified persons and obtain the final result, then the calculation formula of comprehensive similarity is as follows:

[0161] S(i,j)=λ h S h (i,j)+λ g S g (i,j) (11);

[0162] Among them, λ h is the average histogram similarity S h The weight, λ g is the graph similarity S g The weight of the embodiment of the present invention is set to h =0.5,λ g =0.5, in fact, λ h and λ g Can be set arbitrarily, but must satisfy λ h +λ g =1, that is, the preset ratio of the graph similarity and the preset ratio of the average histogram similarity add up to “1”.

[0163] For further information, please refer to Figure 1 S600, respectively calculating the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs, and comparing them with a preset threshold value, if the comprehensive similarity is not less than the preset threshold value, replacing the second relationship subgraph with the first relationship subgraph, otherwise, directly adding the first relationship subgraph to the second relationship subgraph set to update the second relationship subgraph set;

[0164] Specifically, after initializing a second relationship subgraph set and directly adding the first relationship subgraph that appears for the first time to the second relationship subgraph set, the comprehensive similarity of each first relationship subgraph and all second relationship subgraphs is calculated using the above method for calculating the comprehensive similarity, and compared with a preset threshold. If the comprehensive similarity is not less than the preset threshold, the first relationship subgraph replaces the second relationship subgraph; otherwise, the first relationship subgraph is directly added to the second relationship subgraph set to update the second relationship subgraph set.

[0165] For further information, see Fig.11, the step S600 of respectively calculating the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs and comparing them with a preset threshold, if the comprehensive similarity is not less than the preset threshold, replacing the second relationship subgraph with the first relationship subgraph, otherwise, directly adding the first relationship subgraph to the second relationship subgraph set to update the second relationship subgraph set specifically includes:

[0166] S610, respectively calculating the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs, and comparing with the preset threshold each time;

[0167] S620. If the comprehensive similarity is greater than or equal to the preset threshold, the second relationship subgraph is replaced by the first relationship subgraph; if the comprehensive similarity is less than the preset threshold, the first relationship subgraph is directly added to the second relationship subgraph set to update the second relationship subgraph set.

[0168] Specifically, after initializing a second relationship subgraph set and directly adding the first relationship subgraph that appears for the first time to the second relationship subgraph set, traversing and querying the second relationship subgraphs in the second relationship subgraph set, and then calculating the remaining first relationship subgraph sets Z in sequence i The comprehensive similarity between the first relationship subgraph in and all the second relationship subgraphs in the global subgraph set Z (the second relationship subgraph set) is calculated, and each time it is compared with the preset threshold (for example, 0.9):

[0169] If the comprehensive similarity is greater than or equal to the preset threshold, the first relationship subgraph will replace the second relationship subgraph; if the comprehensive similarity is less than the preset threshold, the first relationship subgraph will be directly added to the second relationship subgraph set to update the second relationship subgraph set, that is, for highly similar relationship subgraphs (comprehensive similarity values ​​are greater than 0.9), they can be considered as repeated characters and replaced. The reason for replacement is to update the latest status of the character to avoid the decrease in similarity of the same person over time. For relationship subgraphs with low similarity, they are regarded as new characters added to the global relationship subgraph set, so as to maintain a high similarity with the same detection object in the global relationship subgraph set in real time, that is, to maintain the accuracy of calculating different detection objects in the global relationship subgraph set.

[0170] The algorithm for updating the second relationship subgraph set and performing the headcount counting is as follows:

[0171] Algorithm 2:

[0172] Input: Image sequence

[0173] Output: Total number of people: N

[0174] 1: Using the detection network to detect I t All objects in the and the corresponding bounding box image

[0175] 2: According to Build a relationship diagram

[0176] 3: According to the relationship diagram G i Construct the first relation subgraph set Z for all people i , and get the corresponding bounding box image at the same time Among them, the first relationship subgraph: m is the number of people in the relationship subgraph

[0177] 4: Use Z0 as the initialized global graph set Z (second relationship subgraph set)

[0178] 5: for p = 1, p < = k do / / p traverses from 1 to k, where k is the number of bounding boxes

[0179] 6:i x = Number of subgraphs in Z

[0180] 7:i y =Z p Number of neutron graphs

[0181] 8:S h =0,S g =0

[0182] 9:for q=0,q<=i x do

[0183] 10:for w=0,w<=i y do

[0184] 11: Calculate using Algorithm 1 and B q The average histogram similarity of is the first bounding box image B2 is the second bounding box image

[0185] 12: Calculate according to formula (6)-(8) and g q Graph similarity is the first relation subgraph, g q The second relationship subgraph

[0186] 13:if / / t is the preset threshold, such as t=0.9

[0187] 14: Delete g from Z q and will Add it

[0188] 15: continue

[0189] 16:else

[0190] 17: Join Z

[0191] 18: end for

[0192] 19: end for

[0193] 20: Update Z

[0194] 21: end for

[0195] 22: Calculate the number N of relationship subgraphs in set Z

[0196] 23:return N

[0197] The process of Algorithm 2 is described as follows:

[0198] Initialize an empty set Z. For each person (i) in the image sequence, there is a corresponding bounding box image and a relationship subgraph extracted with the person as the center. Then, perform similarity analysis with the person in Z. If the person (j) is highly similar to the person in Z (the value of the comprehensive similarity is greater than 0.9), remove j from Z and add i to Z. If there is no highly similar person, add j to Z directly. Finally, the number of relationship subgraphs in Z is the total number of people.

[0199] Please continue reading Figure 1 S700: After all the first relationship subgraphs are calculated, the number of the second relationship subgraphs in the updated second relationship subgraph set is calculated to obtain the number of different detection objects in the picture sequence.

[0200] Specifically, the first relationship subgraph set Z is calculated in sequence i The comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs in the global subgraph set Z (the second relationship subgraph set) is calculated, and compared with the preset threshold each time, and the second relationship subgraph set is updated according to the comparison result. After all the first relationship subgraphs are calculated, the number of the second-relationship subgraphs in the updated second relationship subgraph set is calculated, and the number of different detection objects in the image sequence can be obtained, that is, the number of the second-relationship subgraphs in the updated second relationship subgraph set is the number of different detection objects, that is, the accurate total number of people.

[0201] For further information, see Fig.12 The present invention provides a graph relationship network population counting system, the graph relationship network population counting system comprising:

[0202] Bounding box image generation module 100, second relationship subgraph set generation module 200, graph similarity calculation module 300, average histogram similarity calculation module 400, comprehensive similarity solution module 500, preset threshold comparison module 600 and detection object quantity generation module 700;

[0203] The bounding box image generation module 100 is used to use the detection network to predict the input image sequence, obtain the key point position, bounding box and category of all detection objects in the image sequence, and the bounding box size corresponding to the key point, and obtain the bounding box image of the image sequence according to the key point position, the category and the bounding box size; the second relationship subgraph set generation module 200 is used to construct the corresponding first relationship subgraph according to the bounding box of the detection object, and after obtaining the first relationship subgraph set of all detection objects, initialize the second relationship subgraph set and query the first relationship subgraph, and directly add the first relationship subgraph that appears for the first time to the second relationship subgraph set; the graph similarity calculation module 300 is used to respectively normalize and iteratively calculate the correlation between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlation between adjacent edges in the first relationship subgraph and the second relationship subgraph, to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph; the average histogram similarity The similarity calculation module 400 is used to calculate the histogram similarity of the first bounding box image and the second bounding box image on three channels respectively, and calculate the average histogram similarity of the first bounding box image and the second bounding box image; the comprehensive similarity solving module 500 is used to calculate the comprehensive similarity of the first relationship subgraph and the second relationship subgraph according to the image similarity and the average histogram similarity; the preset threshold comparison module 600 is used to calculate the comprehensive similarity of each of the first relationship subgraphs and all the second relationship subgraphs respectively, and compare with the preset threshold. If the comprehensive similarity is not less than the preset threshold, the first relationship subgraph replaces the second relationship subgraph; otherwise, the first relationship subgraph is directly added to the second relationship subgraph set to update the second relationship subgraph set; the detection object number generation module 700 is used to calculate the number of the second relationship subgraphs in the updated second relationship subgraph set after all the first relationship subgraphs are calculated, so as to obtain the number of different detection objects in the image sequence.

[0204] Specifically, first, the detection network (CenterNet network) is used to detect the input image sequence Predictions are made to detect all the detected objects in each picture and obtain a bounding box corresponding to each detected object: (c is the category, and the bounding box contains the key point position and category information of the detected object) and the size of the bounding box corresponding to the key point: Then, the bounding box image of the picture sequence is obtained according to the key point position, the category and the bounding box size: Then, the corresponding relationship graph is constructed based on the bounding box: And construct the first relationship subgraph of all detected objects according to the relationship graph: At the same time, the first relation subgraph set of all detected objects is obtained: Z i ; Secondly, initialize an empty second relationship subgraph set Z, and query all the first relationship subgraphs in the first relationship subgraph set, and then directly add the first relationship subgraph that appears for the first time to the second relationship subgraph set as the initial second relationship subgraph in the second relationship subgraph set Z; Next, calculate the correlation between neighboring nodes in the first relationship subgraph, for example and the correlations between adjacent nodes in the second relation subgraph, e.g. The node similarity between the first relationship subgraph and the second relationship subgraph is calculated in this way: Then the node similarity is iteratively calculated to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph: S g .

[0205] Furthermore, the histogram similarity of the first bounding box image corresponding to the first relationship subgraph and the second bounding box image corresponding to the second relationship subgraph is calculated on the three channels of R, G, and B respectively: S r , S g and S b , thereby calculating the average histogram similarity between the first bounding box image and the second bounding box image: Then, the comprehensive similarity between the character i in the first relationship subgraph and the character j in the second relationship subgraph can be obtained according to the graph similarity and the average histogram similarity: S(i,j)=λ h S h (i,j)+λ g S g(i, j); finally, respectively calculate the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs, and compare them with a preset threshold value; if the comprehensive similarity is not less than the preset threshold value, replace the second relationship subgraph with the first relationship subgraph; otherwise, directly add the first relationship subgraph to the set of the second relationship subgraphs, until the comprehensive similarities of all the first relationship subgraphs and all the second relationship subgraphs are calculated, and then calculate the number of the second relationship subgraphs in the updated set of the second relationship subgraphs to obtain the number of different detection objects in the image sequence.

[0206] In the present application, after calculating the node similarity between the first relationship subgraph and the second relationship subgraph according to the correlation between adjacent nodes in the first relationship subgraph and the second relationship subgraph, iterative calculation is performed to obtain the graph similarity of the first relationship subgraph and the second relationship subgraph, and then, by calculating the average histogram similarity of the first bounding box image corresponding to the first relationship subgraph and the second bounding box image corresponding to the second relationship subgraph, after solving the comprehensive similarity of the graph similarity and the average histogram similarity, it is compared with a preset threshold, and the second relationship subgraph set is updated accordingly according to the comparison result, so as to obtain the number of different detection objects in the image sequence, thereby efficiently and accurately counting the number of different detection objects in the image.

[0207] For further information, see Fig.13 The present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Fig.13 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0208] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a graph relationship network population counting program 40 is stored on the memory 20, and the graph relationship network population counting program 40 can be executed by the processor 10, thereby realizing the defense method for drone swarms based on soft constraints in the present invention.

[0209] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the soft constraint-based defense method against drone swarms.

[0210] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the device and to display a visual user interface. The components 10-30 of the device communicate with each other via a system bus.

[0211] In one embodiment, when the processor 10 executes the graph relationship network population counting program 40 in the memory 20 , the steps of the graph relationship network population counting method described above are implemented.

[0212] Furthermore, the present invention also provides a computer-readable storage medium, which stores a graph relationship network population counting program, and when the graph relationship network population counting program is executed by a processor, it implements the steps of the graph relationship network population counting method as described above; since the steps of the graph relationship network population counting method are described in detail above, they will not be repeated here.

[0213] In summary, the present invention provides a graph relational network people counting method and related equipment, the method comprising the following steps: using a detection network to predict an input image sequence, obtaining key point positions, bounding boxes and categories of all detection objects in the image sequence, and bounding box sizes corresponding to key points, and obtaining bounding box images of the image sequence according to the key point positions, the categories and the bounding box sizes; constructing a corresponding first relational subgraph according to the bounding boxes of the detection objects, and after obtaining a first relational subgraph set of all detection objects, initializing a second relational subgraph set and querying the first relational subgraph, and directly adding the first relational subgraph that appears for the first time to the second relational subgraph set; respectively normalizing and iteratively calculating the correlations between adjacent nodes in the first relational subgraph and the second relational subgraph, and the correlations between adjacent edges in the first relational subgraph and the second relational subgraph, to obtain the The method comprises the steps of: calculating the graph similarity between the first relationship subgraph and the second relationship subgraph; calculating the histogram similarity of the first bounding box image and the second bounding box image on three channels respectively, and calculating the average histogram similarity of the first bounding box image and the second bounding box image; calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity; calculating the comprehensive similarity between each of the first relationship subgraph and all the second relationship subgraphs respectively, and comparing them with a preset threshold value, if the comprehensive similarity is not less than the preset threshold value, replacing the second relationship subgraph with the first relationship subgraph, otherwise, directly adding the first relationship subgraph to the second relationship subgraph set to update the second relationship subgraph set; when all the first relationship subgraphs are calculated, calculating the number of the second relationship subgraphs in the updated second relationship subgraph set to obtain the number of different detection objects in the image sequence. After calculating the graph similarity between the first relationship subgraph and the second relationship subgraph, and the average histogram similarity between the first bounding box image corresponding to the first relationship subgraph and the second bounding box image corresponding to the second relationship subgraph, the comprehensive similarity of the graph similarity and the average histogram similarity is solved and compared with a preset threshold, and the second relationship subgraph set is updated accordingly according to the comparison result to obtain the number of different detection objects in the image sequence, thereby efficiently and accurately counting the number of different detection objects in the image.

[0214] It is understandable that those skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention, and all these changes or substitutions should fall within the protection scope of the claims attached to the present invention.

Claims

1. A method for counting people in a graph relationship network, characterized in that: The graph relationship network population counting method comprises the following steps: Using the detection network to predict the input image sequence, obtain the key point positions, bounding boxes and categories of all detection objects in the image sequence, and the bounding box sizes corresponding to the key points, and obtain the bounding box images of the image sequence according to the key point positions, the categories and the bounding box sizes; Constructing a corresponding first relationship subgraph according to the bounding box of the detected object, after obtaining a first relationship subgraph set of all detected objects, initializing a second relationship subgraph set and querying the first relationship subgraph, and directly adding the first relationship subgraph that appears for the first time to the second relationship subgraph set; Normalizing and iteratively calculating the correlations between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlations between adjacent edges in the first relationship subgraph and the second relationship subgraph, respectively, to obtain graph similarity between the first relationship subgraph and the second relationship subgraph; Calculating the histogram similarity of the first bounding box image and the second bounding box image on three channels respectively, and calculating the average histogram similarity of the first bounding box image and the second bounding box image; Calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity; Calculate the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs respectively, and compare them with a preset threshold value; if the comprehensive similarity is not less than the preset threshold value, replace the second relationship subgraph with the first relationship subgraph; otherwise, directly add the first relationship subgraph to the second relationship subgraph set to update the second relationship subgraph set; When all the first relationship subgraphs are calculated, the number of the second relationship subgraphs in the updated second relationship subgraph set is calculated to obtain the number of different detection objects in the picture sequence.

2. The method for counting people in a graph relationship network according to claim 1, characterized in that: The step of using the detection network to predict the input image sequence, obtaining the key point positions, bounding boxes and categories of all detection objects in the image sequence, and the bounding box sizes corresponding to the key points, and obtaining the bounding box images of the image sequence according to the key point positions, the categories and the bounding box sizes specifically includes: Compressing the picture sequence in units of step length to obtain a compressed image, using the detection network to predict the key point positions and categories of all the detection objects in the compressed image, and simultaneously obtaining a bounding box; Predicting a corresponding key point heat map according to each of the categories, and splicing all the key point heat maps to obtain a predicted heat map, and then calculating a loss value according to the predicted heat map and a reference heat map of the category to obtain the bounding box size; The bounding box image of the picture sequence is restored according to the key point position, the category and the bounding box size of the compressed image.

3. The method for counting people in a graph relationship network according to claim 2, characterized in that: The step of constructing a corresponding first relationship subgraph according to the bounding box of the detected object, obtaining a first relationship subgraph set of all detected objects, initializing a second relationship subgraph set and querying the first relationship subgraph, and directly adding the first relationship subgraph that appears for the first time to the second relationship subgraph set specifically includes: A relationship graph is constructed by taking the bounding box of the detected object as a central node and other adjacent detected objects as neighbor nodes; Refining the relationship graph, and constructing a relationship subgraph according to the refined relationship graph to obtain the first relationship subgraph set of all the detection objects; After initializing an empty second relationship subgraph set, query the first relationship subgraph in the first relationship subgraph set, and directly add the first relationship subgraph that appears for the first time to the second relationship subgraph set.

4. The method for counting people in a graph relationship network according to claim 1, characterized in that: The step of performing normalization processing and iterative calculation on the correlations between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlations between adjacent edges in the first relationship subgraph and the second relationship subgraph, respectively, to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph specifically includes: After respectively calculating the correlation between adjacent nodes in the first relationship subgraph and the correlation between adjacent nodes in the second relationship subgraph, normalization processing is performed to obtain the node similarity between the first relationship subgraph and the second relationship subgraph, wherein the second relationship subgraph is the first relationship subgraph added to the second relationship subgraph set; After respectively calculating the correlation between adjacent edges in the first relationship subgraph and the correlation between adjacent edges in the second relationship subgraph, normalization processing is performed to obtain edge similarities between adjacent edges in the first relationship subgraph and adjacent edges in the second relationship subgraph; Iteratively calculating the node similarity and the edge similarity to obtain the node similarity matrix and the edge similarity matrix; The corresponding positions in the node similarity matrix are compared to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph.

5. The method for counting people in a graph relationship network according to claim 1, characterized in that: The step of respectively calculating the histogram similarity of the first bounding box image and the second bounding box image on three channels, and calculating the average histogram similarity of the first bounding box image and the second bounding box image comprises: Adjusting the first bounding box image corresponding to the first relationship subgraph and the second bounding box image corresponding to the second relationship subgraph to a preset size, wherein the first bounding box image is the bounding box image corresponding to the first relationship subgraph, and the second bounding box image is the bounding box image corresponding to the second relationship subgraph; Separating the adjusted first bounding box image and the adjusted second bounding box image in three channels respectively, and calculating corresponding histogram similarities in the three channels respectively; An average value of the histogram similarities of the three channels is calculated to obtain an average histogram similarity of the first bounding box image and the second bounding box image.

6. The method for counting people in a graph relationship network according to claim 1, characterized in that: The step of calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity specifically includes: Substituting a preset ratio into the graph similarity and the average histogram similarity for calculation, a comprehensive similarity between the first relationship subgraph and the second relationship subgraph is obtained.

7. The method for counting people in a graph relationship network according to claim 1, characterized in that: The step of respectively calculating the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs and comparing the similarity with a preset threshold, and if the comprehensive similarity is not less than the preset threshold, replacing the second relationship subgraph with the first relationship subgraph, otherwise, directly adding the first relationship subgraph to the second relationship subgraph set to update the second relationship subgraph set specifically includes: respectively calculating the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs, and comparing the similarity with the preset threshold each time; If the comprehensive similarity is greater than or equal to the preset threshold, the first relationship subgraph replaces the second relationship subgraph; if the comprehensive similarity is less than the preset threshold, the first relationship subgraph is directly added to the second relationship subgraph set to update the second relationship subgraph set.

8. A graph relationship network population counting system, characterized in that: The graph relationship network population counting system comprises: A bounding box image generation module is used to predict the input image sequence using the detection network, obtain the key point positions, bounding boxes and categories of all detection objects in the image sequence, and the bounding box sizes corresponding to the key points, and obtain the bounding box image of the image sequence according to the key point positions, the categories and the bounding box sizes; A second relationship subgraph generation module is used to construct a corresponding first relationship subgraph according to the bounding box of the detected object, and after obtaining the first relationship subgraphs of all detected objects, initialize the second relationship subgraph and query the first relationship subgraph, and directly add the first relationship subgraph that appears for the first time to the second relationship subgraph; A graph similarity calculation module, used to respectively normalize and iteratively calculate the correlation between adjacent nodes in the first relationship subgraph and the second relationship subgraph, and the correlation between adjacent edges in the first relationship subgraph and the second relationship subgraph, to obtain the graph similarity between the first relationship subgraph and the second relationship subgraph; an average histogram similarity calculation module, used to calculate the histogram similarity of the first bounding box image and the second bounding box image on three channels respectively, and calculate the average histogram similarity of the first bounding box image and the second bounding box image; A comprehensive similarity solving module, used for calculating the comprehensive similarity between the first relationship subgraph and the second relationship subgraph according to the graph similarity and the average histogram similarity; A preset threshold comparison module is used to calculate the comprehensive similarity between each of the first relationship subgraphs and all the second relationship subgraphs respectively, and compare them with the preset threshold. If the comprehensive similarity is not less than the preset threshold, the second relationship subgraph is replaced by the first relationship subgraph; otherwise, the first relationship subgraph is directly added to the second relationship subgraph set to update the second relationship subgraph set; The detection object quantity generation module is used to calculate the number of the second relation subgraphs in the updated second relation subgraph set after all the first relation subgraphs are calculated, so as to obtain the number of different detection objects in the picture sequence.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a graph relationship network population counting program stored in the memory and executable on the processor. When the graph relationship network population counting program is executed by the processor, the steps of the graph relationship network population counting method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a graph relationship network population counting program, and when the graph relationship network population counting program is executed by a processor, the steps of the graph relationship network population counting method according to any one of claims 1 to 7 are implemented.

Images