A training method and system for a personalized target detection model in smart communities

Abstract

This invention proposes a training method and system for a personalized target detection model in smart communities. The training method includes: capturing a large amount of image data from community surveillance and storing it in a local database, while simultaneously selecting and labeling images for the required tasks; receiving Transformer self-supervised pre-training hyperparameters from a federated server; creating a local training task and performing self-supervised pre-training using unlabeled data from the local database; uploading the network parameters of the model's feature extraction network to the federated server and weighted averaging them to form a global fusion model; each node loading the global fusion model and continuing training with local data until the global fusion model converges; and each community node using the labeled data for personalized fine-tuning to obtain its own target detection model. This invention ensures data privacy and security, significantly reduces the manual input of data labeling, and fully utilizes the massive amount of raw, unlabeled data from the community through self-supervised pre-training.

Description

Technical Field

[0001] This invention belongs to the field of smart community security monitoring, and in particular relates to a training method and system for a personalized target detection model in smart communities. Background Technology

[0002] Community surveillance image analysis is an image processing technology of great interest in the security industry. Simply put, this technology involves identifying objects of interest in images, tracking and analyzing them, promptly detecting abnormal behavior, triggering alarms, and taking other intervention measures. In recent years, with the development of deep learning technology, convolutional neural networks have become the mainstream analysis method. However, due to the complexity of community scenes, the diversity of target types, and the varying image quality, it is difficult to achieve ideal accuracy in surveillance image analysis.

[0003] Vision Transformer, proposed by Google in 2017, is an end-to-end NLP model. This model abandons the traditional sequential structure of RNNs and adopts a self-attention mechanism to enable parallel training and the acquisition of global information. Typically, Transformer models are trained on large text corpora and then fine-tuned on smaller, specific tasks to achieve better computational efficiency and accuracy. Vision Transformer can be seen as a graphical version of the Transformer, directly transferring the standard Transformer model to the image domain with minimal modifications. To adapt the Transformer model to images, the image needs to be segmented into many sub-blocks, and these sub-blocks are combined into linear embedding sequences. These linear embedding sequences are then used as input to the Transformer to simulate word sequence input in NLP. Currently, its accuracy on multiple public datasets has surpassed convolutional neural network methods. However, due to the large number of parameters in the multi-head attention part of the Transformer, it requires a large amount of data for training, making it difficult to achieve optimal performance with insufficient samples.

[0004] Federated learning is a distributed machine learning technique. Its core idea is to train a global model across multiple data sources with local data, building upon virtual fused data by exchanging model parameters or intermediate results without needing to exchange local individual or sample data. This achieves a balance between data privacy and shared computation. Image data from different communities is often not shared for centralized model training due to privacy and security concerns, making federated learning a viable option. However, in the field of smart community object detection, federated learning typically requires a large amount of labeled data to ensure the accuracy and generalization of the federated model.

[0005] Currently, transformer-based object detection algorithms have surpassed convolutional neural network-based algorithms in accuracy, becoming the top object detection method. However, transformer-based object detection algorithms require massive amounts of data for training. While a large number of raw images can be easily obtained from community surveillance, labeling data requires a significant amount of human resources. Summary of the Invention

[0006] To address the above problems, the first aspect of this invention provides a method for training a personalized target detection model for smart communities, applicable to various community nodes, comprising the following steps:

[0007] S1 captures a large amount of image data from community surveillance and stores it in a local database;

[0008] S2, select and label the images required for the task from the local database;

[0009] S3, Receive the self-supervised pre-training hyperparameters of Vision Transformer issued by the federation server. The training hyperparameters are obtained by the federation server after creating the training task.

[0010] S4. After receiving the training hyperparameters, create a local Transformer self-supervised training task and perform self-supervised pre-training using unlabeled data in the local database.

[0011] S5, upload the network parameters of the encoder part of the feature extraction network of the Transformer self-supervised training model of the community node to the federated server;

[0012] S6, Receive the global fusion model issued by the federation server. The global fusion model is obtained by the federation server collecting the model parameters of each community node and then performing a weighted average fusion.

[0013] S7, load the global fusion model and continue training the loaded model using local data;

[0014] S8, loop through S4 to S7 until the global fusion model converges;

[0015] S9, for its own personalized object detection needs, loads the designed object detection algorithm model, uses the encoder part of the global model as the feature extraction network and freezes it, and uses the labeled data to make personalized fine-tuning to obtain its own object detection model.

[0016] Preferably, the specific process of performing self-supervised pre-training using unlabeled data in the local database in step S4 is as follows:

[0017] The input image is divided into multiple non-overlapping image patches, and a portion of each patch is randomly masked. The network employs an asymmetric encoder-decoder structure. The encoder is a pre-trained feature extraction network that extracts features from the unmasked image regions during training. The decoder is a fully connected linear layer that reconstructs the masked regions by combining the features extracted by the encoder. This involves regressing and predicting the pixel values ​​at each masked region location, and pre-training the network by backpropagating the error.

[0018] Preferably, the calculation formula for weighted average fusion in step S6 is as follows:

[0019]

[0020] Where, n k Let n be the number of samples at the k-th node, and n be the total number of samples. For the node model uploaded by node k during the t-th round of fusion, w t This is the global model obtained in this round of fusion. Each parameter of the model is calculated by weighted averaging of the models uploaded by each node. The weight is the ratio of the number of samples in this node to the total number of samples.

[0021] Preferably, the specific process of step S9 is as follows:

[0022] Select the desired object detection algorithm; select the feature fusion network in the object detection algorithm; design the prediction head of the algorithm, including the object classification prediction head and the coordinate regression prediction head; use the encoder part of the global model obtained by federated self-supervised pre-training as the feature extraction network, freeze it, and use the labeled data for training and fine-tuning.

[0023] Preferably, the selected target detection algorithm is any one of Faster R-CNN, Mask R-CNN, or CascadeMask R-CNN; and the selected feature fusion network in the target detection algorithm is any one of FPN, PAFPN, or NASFPN.

[0024] A second aspect of the present invention provides a training device for a personalized target detection model in a smart community. The device includes at least one processor and at least one memory, the processor and the memory being coupled together. The memory stores a computer program. When the processor executes the computer program stored in the memory, the device implements the training method for the personalized target detection model as described in the first aspect.

[0025] A third aspect of the present invention provides a computer-readable storage medium storing a program or instructions that, when executed by a processor, cause a computer to perform a training method for a personalized object detection model as described in the first aspect.

[0026] The fourth aspect of the present invention provides a training system for a personalized target detection model for a smart community, comprising a training device for a personalized target detection model for a smart community as described in the second aspect for several community nodes, and a federated server.

[0027] The beneficial effects of this invention are:

[0028] 1. Knowledge sharing between different communities was achieved while ensuring data privacy and security;

[0029] 2. It significantly reduces the manual input of data annotation and makes full use of the massive amount of raw unlabeled data in the community through self-supervised pre-training, so that it can serve the image analysis of smart communities and achieve a high accuracy rate.

[0030] 3. The pre-trained model obtained through joint training can be applied to various target detection tasks in smart communities, including but not limited to personnel detection, vehicle detection, stray / pet cat and dog recognition, face detection, etc. Deployment and application over a period of time have shown that this invention achieves high-precision personalized target detection. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of the training method for the personalized target detection model in smart communities according to the present invention.

[0033] Figure 2 Flowchart for training image capture;

[0034] Figure 3 Here is a flowchart of the personalized pedestrian detection algorithm for smart communities based on Cascade Mask RCNN in Example 1;

[0035] Figure 4 This is a simplified block diagram of the smart community personalized target detection model training device of the present invention. Detailed Implementation

[0036] The invention will be further described below with reference to specific embodiments.

[0037] Example 1:

[0038] This embodiment combines Figure 1 , Figure 2 , Figure 3 The specific workflow of the training method for the personalized target detection model in smart communities of this invention is described below:

[0039] A large amount of image data is captured from community surveillance and stored in a local database. The differences between frames in the accessed video stream are compared; specifically, a frame is captured at 3-second intervals, and the absolute value of the difference is calculated by subtracting adjacent frames. When the difference exceeds a set threshold, it is considered a moving image, and the captured moving image is written to the local server's hard drive. The specific process is as follows: Figure 2 As shown.

[0040] The system filters and labels images for the required tasks from the local database. For example, each community node filters some images containing people from the local database and labels the filtered images with Labelme, marking the four points on the top, bottom, left, and right of the pedestrian's border.

[0041] The system receives self-supervised pre-training hyperparameters for Vision Transformer from the federated server. The training hyperparameters are obtained by the federated server after creating the training task. The federated server creates its own supervised pre-training task and distributes the training hyperparameters to each community federated node, including the asymmetric encoder-decoder network structure, local training rounds, node communication rounds, batch size, and optimizer.

[0042] After receiving the training hyperparameters, a local Transformer self-supervised training task is created, using unlabeled data from the local database for self-supervised pre-training. The input image is divided into multiple non-overlapping image patches, with a portion randomly covered. The network adopts an asymmetric encoder-decoder structure. The encoder is a pre-trained feature extraction network that extracts features from the uncovered image regions during training. The decoder is a fully connected linear layer that combines the features extracted by the encoder to reconstruct the covered regions, i.e., it performs regression prediction on the pixel values ​​at each covered region location. The network is pre-trained by backpropagating the error.

[0043] The network parameters of the encoder part of the Transformer self-supervised training model feature extraction network in each community node are uploaded to the federated server. After collecting the model parameters from each community node, the federated server performs a weighted average fusion of all parameters in the model to obtain a global fused model, which is then distributed to each community node. The fusion algorithm calculation formula is as follows:

[0044]

[0045] Where, n k Let n be the number of samples at the k-th node, and n be the total number of samples. For the node model uploaded by node k during the t-th round of fusion, w t This is the global model obtained in this round of fusion. Each parameter of the model is calculated by weighted averaging of the models uploaded by each node. The weight is the ratio of the number of samples in this node to the total number of samples.

[0046] The community node loads the global fusion model and continues to train the loaded model using local data, repeating the above process until the joint model converges;

[0047] This embodiment uses the Cascade Mask R-CNN algorithm for pedestrian detection. Of course, Faster R-CNN, Mask R-CNN, and other algorithms can also be used, all of which are applicable to this invention. Figure 3 As shown, the algorithm model consists of four parts: image input, feature extraction, feature fusion, and prediction head. The feature extraction part is the encoder part of the network obtained through federated self-supervised pre-training. The feature fusion network is FPN (Feature Pyramid Network), but PAFPN or NASFPN networks can also be used. The prediction of the detected target includes two parts: a target classification prediction head and a coordinate regression prediction head. During training, the feature extraction network part is frozen, and the training loss functions are cross-entropy loss function and MSE loss function. The pedestrian images labeled in the data processing part are input into the network, and the feature fusion module and prediction head are fine-tuned through backpropagation. Thus, the personalized target detection model for smart communities is obtained.

[0048] Example 2:

[0049] like Figure 4As shown, this invention also provides a training device for a personalized target detection model in a smart community. The device includes at least one processor and at least one memory, as well as a communication interface and an internal bus. The memory stores a computer-executable program. When the processor executes the executable program stored in the memory, the device executes the training method for the personalized target detection model as described in Embodiment 1. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus in the accompanying drawings is not limited to only one bus or one type of bus. The memory may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage device, or a USB flash drive, portable hard drive, read-only memory, disk, or optical disk, etc. The device can be provided as a terminal, server, or other form of device.

[0050] Figure 4 This is a block diagram illustrating an illustrative device. The device may include one or more of the following components: a processing component, a memory, a power supply component, a multimedia component, an audio component, an input / output (I / O) interface, a sensor component, and a communication component. The processing component typically controls the overall operation of the electronic device, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. The processing component may include one or more processors to execute instructions to perform all or part of the steps of the methods described above. Furthermore, the processing component may include one or more modules to facilitate interaction between the processing component and other components. For example, the processing component may include a multimedia module to facilitate interaction between the multimedia component and the processing component.

[0051] Memory is configured to store various types of data to support the operation of electronic devices. Examples of this data include instructions for any application or method used to operate on an electronic device, contact data, phonebook data, messages, pictures, videos, etc. Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0052] A power supply component provides power to various components of an electronic device. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device. A multimedia component includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component includes a front-facing camera and / or a rear-facing camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0053] The audio component is configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. The I / O interface provides an interface between the processing component and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a power button, and a lock button.

[0054] The sensor assembly includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, the sensor assembly can detect the on / off state of the electronic device, the relative positioning of components such as the display and keypad of the electronic device, changes in the position of the electronic device or a component of the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device, and temperature changes of the electronic device. The sensor assembly may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0055] The communication component is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic device can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0056] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0057] Example 3:

[0058] The present invention also provides a non-volatile computer-readable storage medium storing a program or instructions that, when executed by a processor, cause a computer to perform a training method for a personalized target detection model as described in Embodiment 1.

[0059] Specifically, a system, apparatus, or device may be provided equipped with a readable storage medium on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system, apparatus, or device reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the embodiments described above, therefore, the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0060] The aforementioned storage media can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disks or optical disks (such as CD-ROM, CD-R, CD-RW, DVD-20ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tape, etc. The storage media can be any available medium accessible to general-purpose or special-purpose computers.

[0061] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0062] It should be understood that the storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and storage medium can exist as discrete components in a terminal or server.

[0063] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0064] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0065] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0066] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for training a smart community personalized target detection model, applied to each community node, characterized in that, The method comprises the following steps: S1, a large amount of image data is captured from community monitoring and stored in a local database; S2, images required by a task are screened from the local database and labeled; S3, a Vision Transformer self-supervised pre-training hyperparameter issued by a federal server is received, the training hyperparameter being obtained after the federal server creates a training task; S4, after receiving the training hyperparameter, a local Transformer self-supervised training task is created, and unlabeled data in the local database is used for self-supervised pre-training; the specific process is as follows: an input image is divided into a plurality of non-overlapping image blocks, and a part is randomly masked; the network adopts an asymmetric encoder-decoder structure, the encoder is a feature extraction network for pre-training, and feature extraction is performed on an image area that is not masked during the training process; the decoder is a fully connected linear layer, which combines the features extracted by the encoder to reconstruct the masked area, that is, the value of each masked area position pixel is predicted, and the pre-training of the network is realized by back propagation of the error; S5, the network parameter of the encoder part of the Transformer self-supervised training model of the community node is uploaded to the federal server; S6, a global fusion model issued by the federal server is received, the global fusion model being obtained by the federal server after collecting the model parameters of each community node and then performing weighted average fusion; S7, the global fusion model is loaded, and the loaded model is further trained by using local data; S8, steps S4 to S7 are repeated until the global fusion model converges; S9, for the own personalized target detection requirement, a designed target detection algorithm model is loaded, the encoder part of the global fusion model is used as a feature extraction network and is frozen, and the target detection model of the own is obtained by using the data that has been labeled for personalized fine-tuning; the specific process is as follows: selecting a required target detection algorithm; selecting a feature fusion network in the target detection algorithm; designing a prediction head of the algorithm, including a target classification prediction head and a coordinate regression prediction head; using the encoder part of the global fusion model obtained by the federal self-supervised pre-training as a feature extraction network, and freezing the encoder part, and training and fine-tuning by using the data that has been labeled. 2.The method of claim 1, wherein, The calculation formula for weighted average fusion in step S6 is: wherein, is the number of samples of the th node, is the total number of samples of all nodes, is the node model uploaded by the th node in the th round of fusion, is the global fusion model obtained in this round of fusion, and each parameter of the model is calculated by weighted average of the models uploaded by each node, and the weight size is the ratio of the number of samples of the node to the total number of samples. 3.The method of claim 1, wherein: The selected target detection algorithm is any one of Faster RCNN, Mask RCNN or Cascade Mask RCNN; the selected feature fusion network in the target detection algorithm is any one of FPN, PAFPN or NASFPN.

4. A smart community personalized target detection model training device, characterized in that, The device comprises at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer program; when the processor executes the computer program stored in the memory, the device realizes the training method of the personalized target detection model according to any one of claims 1 to 3.

5. A computer readable storage medium, characterized in that, The computer readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to make the computer execute the training method of the personalized target detection model according to any one of claims 1 to 3.

6. A smart community personalized target detection model training system, characterized in that: The intelligent community personalized target detection model training device according to claim 4 comprises a plurality of community nodes, and a federation server.

Images