A method, device and related components for detecting children's independent actions

Abstract

The present invention discloses a method, device and related components for detecting children acting alone, and the method includes: obtaining a first training set, a second training set and a third training set; inputting the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model; introducing a graph convolutional network into the temporary training model to obtain a final training model; inputting the third training set into the final training model for training to obtain a detection model; obtaining video data and performing frame extraction processing on the video data to obtain detection data; inputting the detection data into the detection model to obtain an alarm probability; judging whether the alarm probability reaches a threshold; if the alarm probability reaches the threshold, an alarm is issued; if the alarm probability does not reach the threshold, no alarm is issued. The present invention improves the accuracy and reliability of child lone detection through a multi-module combination of a generative adversarial network, a target detection network and a graph convolutional network.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for detecting children's independent actions and related components. Background Art

[0002] With the rapid development of artificial intelligence Internet of Things (AIoT) technology and the continuous optimization of deep learning algorithms, security management in communities and large-scale property scenarios is gradually moving towards intelligence and refinement. As a core component of the security system, video surveillance plays a vital role in these scenarios. At present, intelligent detection of violations such as motor vehicles occupying the road and electric vehicles parking randomly has been widely used in community management, which not only effectively improves the efficiency of property management, but also significantly enhances the safety and order of the community.

[0003] However, among the many challenges of property safety management, the detection of the specific scenario of "children walking alone" still faces many difficulties. As an open and complex space, the community has a changeable internal environment and frequent crowd movement, and the behavior trajectory and status of children are also highly dynamic. Traditional video surveillance and target detection methods are powerless in dealing with such complex scenarios. Especially when the shadows of children overlap with those of adults, the lighting conditions change, the children's bodies are partially blocked, or the image quality is blurred, traditional methods often have difficulty in accurately identifying and issuing warnings, which greatly reduces the practicality and reliability of the monitoring system.

[0004] Most existing child detection methods rely on a single target detection algorithm, such as the YOLO series, which has achieved impressive results on standard test sets, but its recognition accuracy and robustness are greatly reduced when faced with complex situations in the real world. The limitations of a single algorithm are particularly prominent when children are in complex environments, such as close contact with adults, in motion, or disturbed by light.

[0005] In addition, the current technical system mainly focuses on the analysis of single-frame image data in terms of personnel identification, ignoring the coherence and temporal relationship of behavioral characteristics within continuous time frames. As a dynamic behavior, the essential characteristics of children walking alone are often reflected in a series of continuous actions and state changes. However, the existing detection methods fail to make full use of the temporal information between multiple frames of images, resulting in a high false alarm rate when detecting isolated children's behavior, and failing to provide accurate and timely warning information for property management personnel. Summary of the invention

[0006] The purpose of the present invention is to provide a method, device and related components for detecting children acting alone, aiming to solve the problems of poor accuracy of existing child detection methods.

[0007] In a first aspect, an embodiment of the present invention provides a method for detecting a child acting alone, comprising:

[0008] Acquire a data set and preprocess the data set to obtain a first training set, a second training set, and a third training set;

[0009] Inputting the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model;

[0010] Introduce the graph convolutional network into the temporary training model to obtain the final training model;

[0011] Inputting the third training set into the final training model for training to obtain a detection model;

[0012] Acquire camera data and perform frame extraction processing on the camera data to obtain detection data;

[0013] Inputting the detection data into the detection model to obtain an alarm probability;

[0014] Determining whether the alarm probability reaches a threshold;

[0015] If the alarm probability reaches the threshold, an alarm is issued; if the alarm probability does not reach the threshold, no alarm is issued.

[0016] In a second aspect, an embodiment of the present invention provides a device for detecting a child acting alone, comprising:

[0017] An acquisition unit, used for acquiring a data set and preprocessing the data set to obtain a first training set, a second training set, and a third training set;

[0018] A first training unit is used to input the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model;

[0019] The introduction unit is used to introduce the graph convolutional network into the temporary training model to obtain the final training model;

[0020] A second training unit is used to input the third training set into the final training model for training to obtain a detection model;

[0021] A frame extraction unit, used for acquiring camera data and performing frame extraction processing on the camera data to obtain detection data;

[0022] An input unit, used for inputting the detection data into the detection model to obtain an alarm probability;

[0023] A judging unit, used to judge whether the alarm probability reaches a threshold;

[0024] The alarm unit is used to issue an alarm if the alarm probability reaches a threshold value; and not issue an alarm if the alarm probability does not reach the threshold value.

[0025] In a third aspect, an embodiment of the present invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the child solo action detection method described in the first aspect when executing the computer program.

[0026] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the child individual action detection method described in the first aspect above is implemented.

[0027] The present invention discloses a method, device and related components for detecting children acting alone. The method comprises: obtaining a first training set, a second training set and a third training set; inputting the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model; introducing a graph convolutional network into the temporary training model to obtain a final training model; inputting the third training set into the final training model for training to obtain a detection model; obtaining video data and performing frame extraction processing on the video data to obtain detection data; inputting the detection data into the detection model to obtain an alarm probability; judging whether the alarm probability reaches a threshold; if the alarm probability reaches the threshold, an alarm is issued; if the alarm probability does not reach the threshold, no alarm is issued. The present invention couples the generative adversarial network with the target detection network to achieve more accurate recognition of fuzzy and partially difficult-to-identify objects under complex realistic conditions; by introducing a graph convolutional network to model the spatiotemporal relationship between children and adult targets detected in different time frames, timely judgment and feedback on children walking alone are achieved, the accuracy and reliability of children walking alone detection are improved, potential dangers are avoided in advance, the cost of human patrols is reduced, and the intelligent level of property safety management is improved. The embodiments of the present invention also provide a child acting alone detection device, a computer-readable storage medium, and a computer device, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0029] Figure 1Flow chart of the detection method for children acting alone;

[0030] Figure 2 Schematic diagram of the coupling structure of the generated adversarial network and the target detection network;

[0031] Figure 3 It is a flowchart of the feature processing module based on GCN;

[0032] Figure 4 for Figure 3 Schematic diagram of the structure of the attention module in ;

[0033] Figure 5 The schematic block diagram of a device for detecting children's independent behavior is shown in FIG. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be understood that in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more of the features, integers, steps, operations, elements, components and / or combinations thereof.

[0036] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include plural forms unless the context clearly indicates otherwise.

[0037] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0038] See also Figure 1 This embodiment provides a method for detecting a child acting alone, including:

[0039] S101: Acquire a data set and preprocess the data set to obtain a first training set, a second training set, and a third training set;

[0040] In this embodiment, before obtaining the data set, it is necessary to screen the scenes suitable for children acting alone. Specifically, based on different surveillance cameras in different communities, select the scenes under the camera that are suitable for detecting children acting alone, such as amusement parks, public activity areas, unit building lobbies, and outdoor lawns.

[0041] Then, the corresponding data set is obtained from the camera corresponding to the adaptation scene. For example, the camera corresponding to the adaptation scene is framed to obtain images, and the frame extraction time span is set to one year to capture the various forms of children and adults in various weather, seasons, clothing and backgrounds within a year to improve the generalization ability of the model, and all the frame extraction images are collected to obtain the data set.

[0042] After the data set is obtained, the data set is preprocessed to obtain the first training set, the second training set and the third training set, including: obtaining the data set and screening and cleaning the data set to obtain a processed image data set; then annotating the processed image data set, and using the annotated processed image data set as the first training set; then performing data enhancement processing on the first training set to obtain a second training set.

[0043] The second training set can be obtained by performing data augmentation (such as rotation, scaling, cropping, flipping, etc.) on the first training set. This not only increases the diversity of the data, but also helps the model better generalize to unseen data and reduces the risk of overfitting. Data augmentation is an effective means to improve the robustness and generalization ability of the model.

[0044] In some embodiments, obtaining a data set and screening and cleaning the data set to obtain a processed image data set; annotating the processed image data set, and using the annotated processed image data set as the first training set includes: obtaining a data set and screening and cleaning the data set, such as deleting invalid, low-resolution and damaged images, leaving only n images (i.e., processed image data sets) with clear target objects, complete outlines, high differentiation between children and adults, and no obstructions or defects. Then annotate the n images, with the annotated categories of children and adults, respectively, and record the annotated data set as X1 (i.e., the first training set).

[0045] Using a high-quality, screened and cleaned dataset for training can reduce the amount of useless information that the model needs to process during training, thereby speeding up training. In addition, targeted datasets and accurate annotations can also help the model learn key features faster, further improving training efficiency.

[0046] In this embodiment, in order to improve the recognition ability of the model in various complex real-life scenarios, especially the distinction between children and adolescents, target occlusion, blurred or incomplete target recognition, it is necessary to perform data enhancement processing on the data set X1 to obtain the second training set.

[0047] Specifically, performing data enhancement processing on the first training set to obtain the second training set includes: generating multiple corresponding images by deliberately blocking, cropping, splicing, rotating, flipping, and blurring each image in the first training set, and collecting the multiple images corresponding to all the images in the first training set to obtain the second training set. For example, 5 corresponding images are generated by deliberately blocking, cropping, splicing, rotating, flipping, and blurring each image in X1, so that a total of 5n images are generated corresponding to the n images in X1, and then the 5n image sets are combined and recorded as data set X2.

[0048] Next, you need to capture video clips under the camera corresponding to the adaptation scene, define an event as a collection of all frame-sampling images under a video clip, and then combine all frame-sampling results to infer whether the event is ultimately an alarm or not.

[0049] Specifically, obtaining a data set and preprocessing the data set to obtain a first training set, a second training set, and a third training set also includes: obtaining a data set and intercepting video clips in which only a single child's actions appear throughout the data set, video clips in which both adults and children appear throughout the data set, and video clips in which children, adults, or empty backgrounds do not appear in the video indefinitely, to obtain a video clip set; then performing frame extraction on each video clip in the video clip set to obtain a picture set of each video clip; then defining each picture set as an event, marking all events according to predetermined rules, and obtaining an alarm event set and a non-alarm event set respectively; and then combining the alarm event set and the non-alarm event set to obtain a third training set.

[0050] For example, the duration of the captured video is set to 20 seconds, and a frame is drawn every 5 seconds. An event is a set of 5 pictures (i.e., a set of pictures of each video clip). In order to cope with the detection of children acting alone in various complex real-life scenarios, the video collection mainly includes the following types: videos in which only a single child acts, videos in which both adults and children exist, and videos in which children, adults, or empty backgrounds do not appear indefinitely.

[0051] By capturing specific types of video clips (videos where only a single child appears throughout the entire process, videos where both adults and children appear throughout the entire process, and videos where it is uncertain whether children, adults, or the empty background appear), we can more accurately focus on the scenes that need to be analyzed, thereby improving the relevance and accuracy of the model.

[0052] According to the above collected videos, several different event contents can be formed in the end, and the alarm events are defined as follows:

[0053] The image collections for each video clip consist entirely of individual children;

[0054] The picture set of each video clip consists of a single child + empty background;

[0055] In each picture set of each video clip, there is a child and at most one adult, and it can be clearly seen that the adult and the child are far apart and have no direct connection (passers-by, etc., have no guardianship relationship);

[0056] Therefore, the present embodiment labels all events according to predetermined rules, including: the picture sets of each video clip are composed entirely of single children; or, the picture sets of each video clip are composed of single children + empty background; or, each picture set of each video clip contains one child and at most one adult, and it can be clearly seen that the adults and children are far apart and have no direct connection. Events are marked as alarm events, and except for the above three situations, any other event results are marked as non-alarm events. All alarm events are collected to obtain an alarm event set, and all non-alarm events are collected to obtain a non-alarm event set.

[0057] For example, an event is a collection of 5 pictures. If all 5 frame extraction results are composed of a single child; or, the 5 frame extraction results are composed of a single child + empty background (no more than 2 pictures); or, there is a child in each of the 5 frame extraction results, and at most one adult, and it can be clearly seen that the distance between the adult and the child is far and there is no direct connection (passers-by, etc. have no guardianship relationship); then the event is marked as an alarm event. If it is a situation other than the above three situations, it is marked as a non-alarm event. Finally, there are M events, which are recorded as data set X3 (i.e., the third training set).

[0058] S102: Inputting the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model;

[0059] Specifically, the first training set (i.e. Figure 2 X1 in data data) and the second training set (i.e. Figure 2 X2 in the data data) is input into the model coupled with the generative adversarial network and the target detection network for training, and a temporary training model is obtained, including: inputting the second training set into the generator of the generative adversarial network to obtain generator image data; then inputting the generator image data into the mapping layer to obtain improved image data; then inputting the improved image data and the first training set into the target detection network for training to obtain a temporary training model.

[0060] Inputting the first training set and the second training set into the model coupled with the generative adversarial network and the target detection network for training, the temporary training model also includes:

[0061] The output data of the neck network in the object detection network is input into the discriminator of the generative adversarial network.

[0062] In this embodiment, through the constraint of loss, the generator can learn how to convert or reshape noisy images into clean images under the continuous guidance and correction of the discriminator. The generated results serve as additional inputs to the target detection network, which can help the target detection network to accurately identify partially exposed or blurred children in complex real-life scenarios, thereby reducing the probability of missed detection and false detection.

[0063] More specifically, the target detection network Yolov5 is often difficult to detect or inaccurately detects blurred, occluded, and incomplete targets in complex and diverse real-life scenarios. For this reason, this embodiment integrates the generative adversarial network GAN as a feature enhancement module into the detection network of the target detection network. By repairing, completing, and enhancing unclear and incomplete image features that the target detection network cannot handle well, the target detection network can be made more robust when processing such targets, thereby reducing underreporting and false alarms and improving detection effects.

[0064] like Figure 2 As shown, this embodiment couples the generative adversarial network and the target detection network into a feature enhancement model, which is a dual-stream coupling structure.

[0065] The first stream is the traditional yolov5 feature extraction and detection network, which consists of three parts: the backbone network Backbone, the neck network Neck and the head network Head. First, the deep structure of the backbone network Backbone can capture the main feature information of the input image; secondly, the neck network Neck part layers the features extracted by the Backbone, which adopts a top-down (FPN) and bottom-up (PAN) double-tower structure. The top-down is used to further transmit and enhance the high-level semantic features, and the bottom-up is to transmit some low-level detail information upward, which is helpful for positioning; finally, the Head part outputs the final target box and object label information according to the task orientation under the constraint of the loss function.

[0066] The second stream is the coupling of the Generative Adversarial Network (GAN) and the object detection network (YOLOv5). First, the blurred, occluded, incomplete or incomplete images in the noise dataset X2 (i.e., the second training set) are input into the generator of the Generative Adversarial Network (i.e., Figure 2 In the Generator in the image processing, the generator completes or reconstructs these images by learning to make specific targets clearer; then, after a CBLS mapping layer (CBLS is a combination of convolutional layer Conv, batch normalization BN, linear layer Linear and) the improved image is obtained. (i.e., improving the image data) as an additional input to the target detection network YOLOv5. Here, the main function of the mapping layer is to convert the image output by the generator (i.e., the generator image data) and the clean image X1 (i.e., the first training set) are kept consistent in feature distribution as much as possible, while eliminating The unnecessary interference such as artifacts or noise that may be contained in the image can be further enhanced to further enhance the detection ability of Yolov5 for noisy targets. Finally, the discriminator (i.e. Figure 2 The Discriminator in the target detection network Yolov5 is used to reduce the noise image features output by the Neck end. With clean image features , to encourage the generator to continuously generate "better quality pictures".

[0067] In summary, the collaborative work of the generative adversarial network GAN and the target detection network YOLOv5 enables the generative adversarial network GAN to improve image quality without interfering with the core target detection task of the target detection network YOLOv5, and helps improve the target detection network YOLOv5's ability to recognize noisy images, making the system more robust.

[0068] It should be noted that Figure 2 The Yolov5smodel (i.e. Yolov5s model) refers to a lightweight model in the YOLO (YouOnlyLookOnce) series of target detection algorithms.

[0069] In this embodiment, the generator and discriminator in the generative adversarial network GAN are continuously optimized through adversarial training. The training goal of the generator is to try to deceive the discriminator by minimizing the discriminator's ability to distinguish the features of the generated samples, so that it cannot distinguish the features of the clean samples and the noise samples. The generator receives input from the noise data set X2 and generates an enhanced feature image In order to facilitate the detection network to better identify and detect the target, in this process, the generator will continuously adjust the parameters to optimize the quality of the generated image, which is equivalent to repairing the incompleteness in the reconstructed image or the distortion caused by occlusion and blur. The loss L of the generator G as follows:

[0070]

[0071] Where E represents the expected value of the distribution specified in the subscript; p(X2) represents the distribution of the noisy image data, μ is a constant, and D(G(X2)) is the discriminator's response to the image generated by the generator. The probability of the discrimination result has a value range of [0, 1], where 0 indicates blur and occlusion, and 1 indicates clarity and integrity. The expected value of D(G(X2)) is 1, which means that the image features generated by the generator tend to be clear and complete. The reconstruction loss L reconstruction Can minimize the improved noise image The difference in features from the real clear image X1; represents the L2 norm.

[0072] In this embodiment, the discriminator improves the recognition ability by maximizing the log-likelihood probability of its output, and receives the noise image features in the noise dataset X2 And the final features of the image in the clean dataset X1 By providing gradient information through adversarial loss, the generator is guided to continuously generate more optimized feature expressions for blurred or incomplete targets in the image, thereby improving the model's ability to distinguish targets in various complex scenarios in reality (such as blur, partial occlusion, distinction between teenagers and children, etc.). The loss function L of the discriminator is D is the cross entropy loss for the two categories, and is calculated as follows:

[0073]

[0074] Among them, p(X1) represents the distribution of clean image data, and They represent the probabilities of the discriminator's discrimination results for the features of the real clear image and the improved noisy image features, respectively. Their expected values ​​are 1 and 0, respectively. This means that the discriminator is constantly maximizing the feature difference between the clean image and the noisy image, which forms a confrontation with the reconstruction loss and forces the generator to continuously improve the output image quality.

[0075] Among them, the probability of the discriminator's discrimination result for the real clear image feature is 1, indicating that the discriminator determines that this is the feature of a clear and complete image; the probability of the discriminator's discrimination result for the improved noise image feature is 0, indicating that the discriminator determines that this is the feature of a blurred, incomplete, blocked or incomplete image.

[0076] Furthermore, through the coupled training of the above generative adversarial network GAN and the object detection network YOLOv5, the accuracy and robustness of the model in detecting children's individual actions in real complex scenes can be effectively improved. Finally, the output of the object detection network Yolov5 is the detection results of children and adults in clean images and noisy images, including detection boxes, category labels and confidence. The loss function L1 of the overall framework is as follows:

[0077]

[0078] L cls= -ylog(p)-(1-y)log(1-p)

[0079] L obj =-y obj log(p obj )-(1-y obj )log(1-p obj )

[0080] Among them, the loss of L1 consists of 5 parts, including the bounding box regression loss L bbox , target classification loss L cls and confidence loss L obj are the three losses of the target detection network Yolov5. bbox b and b gt are the center point of the predicted box and the center point of the real box, respectively. c is the diagonal length of the smallest bounding box containing the predicted box and the real box. ρ is the Euclidean distance between the center points. α is the balance factor. ν is a measure of the aspect ratio of the predicted box. IOU represents the intersection-over-union ratio of the predicted box and the real box. L cls Where y is the true category label, which takes 0 or 1 (0 represents a child and 1 represents an adult in this model), and p is the probability that the model predicts that the detection box belongs to a certain category; L obj middle y obj is the true confidence label (1 means there is an object in the box, 0 means there is no object), p obj is the confidence of the model prediction, that is, the probability that the model believes that there is an object in the box. It is worth noting that the image annotation is only performed in the dataset X1, and the annotated X1 is then used to obtain the dataset X2 through data augmentation. Therefore, X2 has its own annotation information, so when GAN and Yolov5 are jointly trained on X2, it has its own groundtruth.

[0081] S103: Introducing the graph convolutional network into the temporary training model to obtain the final training model;

[0082] The model trained by the joint network of Yolov5 and GAN in this embodiment can only have a relatively robust recognition ability for a single static image, but it cannot capture the dynamic change information of children and the surrounding environment in real scenes. For this reason, the graph convolution network GCN is introduced as a further post-processing of the temporary training model. By constructing parallel multi-frame images, it can effectively integrate the target spatial relationship of the single-frame image and the time series relationship of the multi-frame image, and finally form a comprehensive representation vector of the child's relative position and movement changes in a dynamic time window, and then determine whether an alarm is needed through a decision-making mechanism. Such a design enhances the transmission of multi-frame information in the event and the model's ability to understand the overall event, so that the model does not only rely on the detection of a single frame, and ensures the reliability and accuracy of the final judgment result.

[0083] S104: Inputting the third training set into the final training model for training to obtain a detection model;

[0084] Specifically, the third training set is input into the final training model for training, and the detection model is obtained, including: the third training set is input into the temporary training model to obtain the target in each picture; then each target in each picture is cropped and the cropped target is regarded as a node, and the information of each node is linearly mapped through a fully connected network to obtain a node vector set for each picture; then the node vector set is input into the graph convolution network to integrate the information of a single frame to obtain an integrated information vector of each single frame; then the integrated information vectors of each single frame are fused with a temporal relationship dependent on the attention mechanism to obtain a global feature representation vector; then the global feature representation vector is activated through a sigmoid function to obtain the probability of an event alarm; then training is performed based on the cross entropy loss between the probability of the event alarm and the true event label to obtain a detection model.

[0085] The third training set is input into the temporary training model for preliminary target detection, which ensures the accuracy and pertinence of subsequent processing. Next, each target in the image is cropped and processed as a node, which helps to refine the problem to a specific target level and reduce the interference of background information. The information of a single frame is integrated using a graph convolutional network. This technology can capture the complex relationship between nodes and enhance the model's understanding of the interaction between targets in the image. In addition, the integrated information of each single frame is fused through the attention mechanism to rely on the temporal relationship. This step not only considers the spatial relationship within the image, but also the temporal continuity, which is particularly important for processing video or time series data.

[0086] More specifically, the input of the feature processing module based on the graph convolutional network GCN is the data set X3 (i.e., the third training set). The picture set of each event in the data set X3 is input into the model coupled with the generative adversarial network and the target detection network to capture the targets contained therein (i.e., Figure 3 The output detect_out in can be a child or an adult), and then each target detected in each picture is cropped (i.e. Figure 3 After crop truncate in , they are all regarded as a node, and then the node (target) information is linearly mapped through a fully connected network FCN to obtain the vector embedding of each node (target). The vector embedding will be used to construct GCN.

[0087] For example, there are M events in X3, and each event consists of 5 pictures extracted from a video frame. The 5 pictures of each event are passed through a model coupled with a generative adversarial network and a target detection network to capture the targets contained therein. Then, each target detected in each picture is cropped and treated as a node. The node (target) information is linearly mapped through a fully connected network FCN to obtain the vector of each node (target).

[0088] In this embodiment, the construction of the graph convolutional network GCN is divided into the integration of single-frame image information and the transmission of multi-frame image information. A single-frame image corresponds to a layer of the graph convolutional network GCN, whose nodes are determined by the objects in the image, and whose edges are determined by the relative positions, relative sizes, and label categories between the objects; the transmission of multi-frame images integrates the output dependency time sequence relationship of each layer of single-frame images through the attention mechanism (i.e. Figure 3 final_tensor1 and final_tensor2 in ), and finally obtain the global feature representation vector of the event graph (such as Figure 3 In general, the introduction of the graph convolutional network (GCN) enhances the model's understanding of the situation. The construction of nodes and edges in a single-frame graph enables the model to understand the distribution and relationship of targets in space. The information transmission of multi-frame graphs provides a dynamic perspective for event analysis through the dimension of time. The combination of the two improves the effectiveness of model reasoning and provides a solid basis for whether to issue an alarm in the end.

[0089] Next, the final output of the graph convolutional network GCN is the global feature representation vector final_tensor5. After the global feature representation vector final_tensor5 is activated by the sigmoid function, the probability P of the event alarm is obtained. alert Then, training is performed based on the cross entropy loss between the probability of event alarm and the true event label y to obtain a detection model. The cross entropy loss between the probability of event alarm and the true event label y is performed to make the alarm probability of the three events defined in step S102 tend to 1, while the alarm probability of other events tend to 0.

[0090] In this embodiment, the node vector set is input into the graph convolutional network to integrate the information of the single-frame graph, and the integrated information of each single-frame graph is obtained, including: constructing a degree matrix and an adjacency matrix according to the connection between each node; then normalizing the adjacency matrix by left-multiplying and right-multiplying the adjacency matrix by the inverse matrix of the degree matrix; then multiplying the normalized adjacency matrix and the node vector set of each single-frame graph to obtain the integrated information vector of each single-frame graph.

[0091] Through graph convolutional networks, the information of each node in the graph can be efficiently integrated. This integration not only takes into account the characteristics of the node itself, but also the connection relationship between the nodes, so as to capture richer graph structure information. At the same time, the normalization of the adjacency matrix (using the inverse matrix of the degree matrix for left and right multiplication) helps to balance the connection strength of different nodes in the graph, avoiding some high-degree nodes from occupying too much weight in the process of information propagation, thereby improving the robustness and generalization ability of the model.

[0092] Specifically, for each frame of each event in the dataset X3, Yolov5+GAN is used for target detection, and each detected target is regarded as a node. Assume that in frame t i N i targets (1≤i≤5), then the node set of a single-frame graph can be expressed as: Node (target) in, represents the target category label, They represent the normalized center horizontal coordinate, vertical coordinate, width and height of the j-th target rectangular box, respectively. Represents the target category label confidence level.

[0093] Then, each node information is linearly mapped through a fully connected layer, and then the frame t is obtained. i The embedding set of all nodes in in Representation Node The corresponding eigenvector, R is a real number, and C is the dimension of the eigenvector. To simplify the description, the symbols in the following formulas are all without the right superscript. For example, the node Remove the upper right label as node v j ,node Remove the upper right label as node v i , target category label Remove the upper right label to get the target category label c j .

[0094] Furthermore, the degree matrix D in the graph convolutional network GCN is defined as a diagonal matrix. The elements on the diagonal are the degrees of each node. The degree of a node represents the number of edges associated with the node. The specific form is as follows:

[0095]

[0096] Among them, the element d in the matrix jj (d jj Located in 11 and Between) indicates that the node v j Related (including v j The number of edges of itself), the degree matrix D is N i Order square matrix (N i is the number of nodes). In particular, when N i = 0, the degree matrix is ​​defined as a zero matrix; when N i When =1, the degree matrix is ​​a first-order unit matrix.

[0097] The adjacency matrix A represents the relationship between nodes and is a symmetric N i order square matrix, when N i = 0, the adjacency matrix is ​​defined as a zero matrix; when N i =1, the adjacency matrix is ​​a square matrix of order 1. In each layer of the graph convolutional network GCN, information is connected through the adjacency matrix, which is specifically defined as follows:

[0098] where a ij =a ji ;

[0099] Among them, a ij (a ij Located in a 11 and Between) represents two nodes v i and v j The weight of the edge between the two targets is considered here, including the spatial distance between the two targets, the target size, and the target category, which may affect the alarm probability. ij The specific definitions are as follows:

[0100]

[0101] in, is node v i and v j The Euclidean distance between i and i Respectively represent the node v i The normalized horizontal and vertical coordinates of the center of the rectangular box, x j and j Respectively represent the node v j The normalized horizontal and vertical coordinates of the center of the rectangular box; σ is a parameter that controls the influence of spatial distance on the weight; s i and j They are node v i and v j The area of ​​the rectangular frame is obtained by the width w of the respective frame × the height h of the frame; α c is the category weight, which depends on the node category. The specific values ​​are as follows:

[0102]

[0103] Among them, the subscript is childs, which means the node v i Category c i and node v j Category c j They are all children. The subscript is adults, which means node v i Category c i and node v j Category c j They are all adults. The subscript is mixed, indicating that the node v i Category c i and node v j Category c j They are adults and children or children and adults.

[0104] In the above formula, the child’s own weight α child >The weight α among children childs >The weight α between adults and children mixed >The weight of the adult himself α adult >Weight among adults α adults , which means increasing the impact of children on the alarm and reducing the impact of adults on the alarm, which helps to improve the accuracy of the model judgment and avoid false alarms. ij The formula also shows that: the greater the distance between two nodes, the greater the weight, which means that the correlation between the two target objects is smaller, and the greater the probability of an alarm; the greater the area difference between two nodes, the smaller the weight, which means that the child may be under the care of an adult, and the smaller the probability of an alarm.

[0105] In this embodiment, the output of the single-layer graph of the graph convolutional network GCN is a combination of the degree matrix D, the adjacency matrix A and the node feature matrix X.

[0106] First, use D -1 AD -1 Normalizing the adjacency matrix A and inverting D can make low-degree nodes have greater weight in information propagation, while the role of high-degree nodes in information propagation is weakened. This standardized strategy is essentially a weighted average, which can promote more balanced information propagation, help smooth the graph structure, and avoid the risk of gradient explosion or disappearance caused by vectors being too dense or sparse due to degree imbalance in GCN training. In reality, this also meets the requirements of child solo action detection, that is, the higher the degree of the node, the more adults / children are associated with it, the less suitable it is to issue an alarm, and the weight of the node needs to be reduced.

[0107] Among them, a low-degree node indicates that the number of edges associated with the node is small, and a high-degree node indicates that the number of edges associated with the node is large.

[0108] Next, by multiplying the normalized adjacency matrix A and the node feature matrix X (i.e., the node vector set), we can get the feature sum vector including the node itself and all its neighbors. This has taken into account the influence of edge connections on the information interaction between nodes. The existence of edges allows GCN to update the state of the current node through the information of neighboring nodes, thereby improving the node feature representation and helping the model better understand the importance of the target in the context. In general, the specific formula for GCN single-frame graph information integration is as follows:

[0109] Z = f(D, X, A) = ReLU(D -1 AD -1 XW)

[0110] Where W∈R C×1 is the weight matrix, ReLU is the activation function, let is the output representation vector of a single layer graph.

[0111] Next, multiple consecutive frames of each event in dataset X3 are used as sub-representatives of the event, and each frame is regarded as a layer of GCN. Based on the attention mechanism, the overall representation of the entire event is finally obtained. The structure of the temporal attention module is as follows: Figure 4 As shown, I1~I5 represent The result obtained after the attention mechanism is calculated as follows:

[0112]

[0113] Among them, W q , W k and W v All is the weight matrix, q i , k i and v i Corresponding to query, key and value, I i That is, final_tensor(i), It is the output of the entire GCN, that is, the global feature representation vector of the event. Then I5 will pass through the sigmoid activation function to obtain the alarm probability P of the event. alert , as shown below:

[0114] P alert =Sigmoid(I5·W out +b out )

[0115] in, is the weight vector, b out ∈R 1×1 is the bias scalar, P alert ∈[0, 1] is a real probability. Finally, the obtained P alert Make cross entropy loss L with the real event label y alert , as shown below:

[0116] L alert =-(ylog(P alert )+(1-y)log(1-P alert ))

[0117] Wherein, y=1 indicates that the event generates an alarm, and y=0 indicates that the event generates no alarm.

[0118] The multi-frame processing method of this embodiment is based on the attention mechanism, so that the final representation of each frame not only depends on the characteristics of the current frame, but also refers to the information of the previous frame. This helps the model to conduct a progressive and comprehensive analysis of the changes in the behavior of children or adults within a dynamic time window, avoiding the contingency of judging only by a single frame image, and making the final judgment result more convincing.

[0119] S105: Acquire the video data and perform frame extraction processing on the video data to obtain detection data;

[0120] In this embodiment, the detection model uses accuracy P, recall R and mean average precision (MAP) as indicators for evaluating model performance during training. After the model training is completed, the detection model is deployed on the camera of the real community after quantization.

[0121] Next, the camera data is acquired and frame extraction is performed on the camera data to obtain the detection data, mainly by imitating the construction method of the third training set mentioned above.

[0122] For example, real-time frame sampling detection is performed on the content captured by the camera, and 5 frames are continuously sampled in a cycle of 20 seconds to obtain detection data.

[0123] S106: Inputting the detection data into the detection model to obtain an alarm probability;

[0124] S107: Determine whether the alarm probability reaches a threshold;

[0125] When the alarm probability reaches the threshold, step S108 is executed, and if the alarm probability does not reach the threshold, step S109 is executed. For example, if the threshold is set to 0.8, if the alarm probability is 0.5, step S109 is executed, and if the alarm probability is 0.9, step S108 is executed.

[0126] S108: If the alarm probability reaches a threshold, an alarm is issued;

[0127] It should be noted that the alarm in this embodiment can be sent to the community command center for manual review and then dispatched for processing, or it can be sent directly to specific logistics personnel.

[0128] S109: If the alarm probability does not reach the threshold, no alarm is issued.

[0129] This embodiment greatly improves the accuracy and reliability of child lone detection by combining multiple modules including target detection network YOLOv5, generative adversarial network GAN and graph convolutional network GCN. On the one hand, the model can more accurately identify targets and distinguish categories for blurred, occluded and incomplete images obtained by extracting frames in complex real-life scenarios; on the other hand, the model can focus more on children and comprehensively analyze children's behavioral changes in combination with dynamic time series information, reducing false positives and missed positives. In summary, this method is suitable for large-scale community property management scenarios and provides a practical intelligent monitoring method for child danger prevention in smart properties.

[0130] See also Figure 5 This embodiment provides a child individual action detection device 500, comprising:

[0131] An acquisition unit 501 is used to acquire a data set and preprocess the data set to obtain a first training set, a second training set, and a third training set;

[0132] A first training unit 502 is used to input the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model;

[0133] An introduction unit 503 is used to introduce the graph convolutional network into the temporary training model to obtain a final training model;

[0134] A second training unit 504 is used to input the third training set into the final training model for training to obtain a detection model;

[0135] A frame extraction unit 505 is used to obtain the camera data and perform frame extraction processing on the camera data to obtain detection data;

[0136] An input unit 506, used to input the detection data into the detection model to obtain an alarm probability;

[0137] A judging unit 507 is used to judge whether the alarm probability reaches a threshold;

[0138] The alarm unit 508 is configured to generate an alarm if the alarm probability reaches a threshold value, and not generate an alarm if the alarm probability does not reach the threshold value.

[0139] Furthermore, the acquisition unit 501 further includes:

[0140] A screening subunit is used to obtain a data set and screen and clean the data set to obtain a processed image data set;

[0141] a labeling subunit, used to label the processed image data set, and use the labeled processed image data set as a first training set;

[0142] The enhancement subunit is used to perform data enhancement processing on the first training set to obtain a second training set.

[0143] Furthermore, the acquisition unit 501 further includes:

[0144] The interception subunit is used to obtain a data set and intercept video clips in which only a single child appears throughout the data set, video clips in which both adults and children appear throughout the data set, and video clips in which neither children, adults nor empty backgrounds appear in the video, to obtain a video clip set;

[0145] A frame extraction subunit, configured to perform frame extraction on each video segment in the video segment set to obtain a picture set of each video segment;

[0146] An event definition subunit, used to define each of the picture sets as an event, mark all events according to a predetermined rule, and obtain an alarm event set and a non-alarm event set respectively;

[0147] The combining subunit is used to combine the alarm event set and the non-alarm event set to obtain a third training set.

[0148] Furthermore, the first training unit 502 further includes:

[0149] A data acquisition subunit, used for inputting the second training set into the generator of the generative adversarial network to obtain generator image data;

[0150] A data input subunit, used for inputting the generator picture data into the mapping layer to obtain improved picture data;

[0151] A temporary training subunit is used to input the improved image data and the first training set into the target detection network for training to obtain a temporary training model.

[0152] Furthermore, the first training unit 502 further includes:

[0153] The intermediate layer data input subunit is used to input the output data of the neck network in the target detection network into the discriminator of the generative adversarial network.

[0154] Furthermore, the second training unit 504 further includes:

[0155] A target acquisition subunit, used for inputting the third training set into a temporary training model to obtain targets in each picture;

[0156] A linear mapping subunit, used to crop each object in each picture and treat the cropped object as a node, and linearly map the information of each node through a fully connected network to obtain a node vector set for each picture;

[0157] An information integration subunit, used for inputting the node vector set into the graph convolution network to perform information integration of single-frame graphs, and obtaining integrated information vectors of each single-frame graph;

[0158] The fusion subunit is used to fuse the integrated information vectors of each single frame image with the time-dependent relationship through the attention mechanism to obtain the global feature representation vector;

[0159] An activation subunit, used to activate the global feature representation vector through a sigmoid function to obtain the probability of an event alarm;

[0160] The cross entropy loss subunit is used to perform training based on the cross entropy loss between the probability of the event alarm and the real event label to obtain a detection model.

[0161] Furthermore, the information integration subunit includes:

[0162] A construction subunit is used to construct a degree matrix and an adjacency matrix according to the connections between each node;

[0163] A normalization subunit, configured to normalize the adjacency matrix by left-multiplying and right-multiplying the adjacency matrix by the inverse matrix of the degree matrix;

[0164] The multiplication subunit is used to multiply the normalized adjacency matrix and the node vector set of each single-frame graph to obtain the integrated information vector of each single-frame graph.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-mentioned devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0166] The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed, the method provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0167] The present invention also provides a computer device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method provided in the above embodiment can be implemented. Of course, the computer device may also include various network interfaces, power supplies and other components.

[0168] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0169] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variations thereof are intended to cover non-exclusive.

[0170] Inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for detecting children acting alone, characterized in that: include: Acquire a data set and preprocess the data set to obtain a first training set, a second training set, and a third training set; Inputting the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model; Introduce the graph convolutional network into the temporary training model to obtain the final training model; Inputting the third training set into the final training model for training to obtain a detection model; Acquire camera data and perform frame extraction processing on the camera data to obtain detection data; Inputting the detection data into the detection model to obtain an alarm probability; Determining whether the alarm probability reaches a threshold; If the alarm probability reaches the threshold, an alarm is issued; if the alarm probability does not reach the threshold, no alarm is issued; The step of inputting the third training set into the final training model for training to obtain the detection model comprises: Inputting the third training set into a temporary training model to obtain targets in each picture; Each target in each picture is cropped and the cropped target is regarded as a node, and the information of each node is linearly mapped through a fully connected network to obtain a node vector set for each picture; Inputting the node vector set into the graph convolutional network to perform information integration of single-frame graphs to obtain integrated information vectors of each single-frame graph; The integrated information vectors of each single frame are fused with the temporal relationship through the attention mechanism to obtain the global feature representation vector. The global feature representation vector is activated by a sigmoid function to obtain the probability of event alarm; The detection model is obtained by training based on the cross entropy loss between the probability of the event alarm and the real event label; The step of inputting the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model comprises: Inputting the second training set into the generator of the generative adversarial network to obtain generator image data; Inputting the generator picture data into a mapping layer to obtain improved picture data; Inputting the improved image data and the first training set into the object detection network for training to obtain a temporary training model; The step of inputting the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model further includes: The output data of the neck network in the target detection network is input into the discriminator of the generative adversarial network.

2. The method for detecting children acting alone according to claim 1, characterized in that: The acquiring a data set and preprocessing the data set to obtain a first training set, a second training set, and a third training set comprises: Acquire a data set and perform screening and cleaning on the data set to obtain a processed image data set; Annotating the processed image data set, and using the annotated processed image data set as a first training set; Perform data enhancement processing on the first training set to obtain a second training set.

3. The method for detecting children acting alone according to claim 1, characterized in that: The acquiring a data set and preprocessing the data set to obtain a first training set, a second training set, and a third training set further comprises: Obtain a data set and extract video clips in which only a single child appears throughout the data set, video clips in which both an adult and a child appear throughout the data set, and video clips in which neither a child, an adult, nor an empty background appears in the video, to obtain a video clip set; Performing frame extraction processing on each video segment in the video segment set to obtain a picture set of each video segment; Each of the picture sets is defined as an event, and all events are marked according to a predetermined rule to obtain an alarm event set and a non-alarm event set respectively; The alarm event set and the non-alarm event set are combined to obtain a third training set.

4. The method for detecting children acting alone according to claim 1, characterized in that: The step of inputting the node vector set into the graph convolution network to integrate the information of the single-frame graph to obtain the integrated information vector of each single-frame graph includes: Construct the degree matrix and adjacency matrix based on the connections between nodes; Normalizing the adjacency matrix by left-multiplying and right-multiplying the adjacency matrix by the inverse matrix of the degree matrix; The normalized adjacency matrix is ​​multiplied by the node vector set of each single-frame graph to obtain the integrated information vector of each single-frame graph.

5. A device for detecting children's independent behavior, characterized in that: include: An acquisition unit, used for acquiring a data set and preprocessing the data set to obtain a first training set, a second training set, and a third training set; A first training unit is used to input the first training set and the second training set into a model coupled with a generative adversarial network and a target detection network for training to obtain a temporary training model; The introduction unit is used to introduce the graph convolutional network into the temporary training model to obtain the final training model; A second training unit is used to input the third training set into the final training model for training to obtain a detection model; A frame extraction unit, used for acquiring camera data and performing frame extraction processing on the camera data to obtain detection data; An input unit, used for inputting the detection data into the detection model to obtain an alarm probability; A judging unit, used to judge whether the alarm probability reaches a threshold; An alarm unit, configured to generate an alarm if the alarm probability reaches a threshold; If the alarm probability does not reach the threshold, no alarm is issued; The second training unit comprises: A target acquisition subunit, used for inputting the third training set into a temporary training model to obtain targets in each picture; A linear mapping subunit, used to crop each object in each picture and treat the cropped object as a node, and linearly map the information of each node through a fully connected network to obtain a node vector set for each picture; An information integration subunit, used for inputting the node vector set into the graph convolution network to perform information integration of single-frame graphs, and obtaining integrated information vectors of each single-frame graph; The fusion subunit is used to fuse the integrated information vectors of each single frame image with the time-dependent relationship through the attention mechanism to obtain the global feature representation vector; An activation subunit, used to activate the global feature representation vector through a sigmoid function to obtain the probability of an event alarm; A cross entropy loss subunit, used for training based on the cross entropy loss between the probability of the event alarm and the real event label to obtain a detection model; The first training unit also includes: A data acquisition subunit, used for inputting the second training set into the generator of the generative adversarial network to obtain generator image data; A data input subunit, used for inputting the generator picture data into the mapping layer to obtain improved picture data; A temporary training subunit, used for inputting the improved image data and the first training set into the object detection network for training to obtain a temporary training model; The first training unit also includes: The intermediate layer data input subunit is used to input the output data of the neck network in the target detection network into the discriminator of the generative adversarial network.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting children acting alone according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to perform the child's independent action detection method according to any one of claims 1 to 4.

Images