A multimodal data analysis method and system based on federated learning

Abstract

The present invention discloses a multimodal data analysis method and system based on federated learning. The multimodal data analysis method based on federated learning includes: obtaining environmental images and audio information; obtaining a motion trajectory classifier; obtaining the motion trajectory of a suspicious moving object; extracting the trajectory features of the motion trajectory of the suspicious object; obtaining audio information in the time period when the motion trajectory of the suspicious object is captured; extracting audio features in the audio information; fusing the trajectory features and the audio features to obtain fused features; inputting the fused features into the motion trajectory classifier to obtain a classification label; and generating an alarm signal when the classification label is a dangerous classification label. The present application can learn the motion trajectory classifier by means of federated learning without aggregating the data sets possessed by each communication tower to a central cloud server, thereby achieving accurate classification of the classifier when each communication tower has a small data set.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a multimodal data analysis method based on federated learning and a communication tower system. Background Art

[0002] Most communication towers are placed in open areas where various people or animals may roam back and forth. At this time, some people may be the maintainers of the communication towers and come to maintain the communication towers. Some people may be wandering around the communication towers or approaching the communication towers out of curiosity, such as children. Some people may want to steal equipment from the communication towers. At this time, it is necessary to monitor the situation around the communication towers and deal with them differently according to different situations.

[0003] In the prior art, most communication towers are equipped with video cameras, but these cameras are only capable of video surveillance and cannot automatically determine various situations and how to deal with each situation. Instead, they can only transmit video information to a central control room for users to make manual judgments. However, on the one hand, this wastes manpower and requires users to monitor all the time. On the other hand, in many cases, a user needs to monitor many videos alone and cannot concentrate on the situation near a communication tower. By the time a user notices something, danger may have already occurred.

[0004] However, in many cases, pedestrians or animals near the communication tower may just be strolling around without any purpose, while in other cases, thieves may want to steal equipment on the communication tower. In this case, a method is needed to report the incident to the police.

[0005] However, in the prior art, false alarms often occur, that is, the true intention of a pedestrian is not recognized, or an animal is mistakenly judged as a pedestrian, thereby generating an alarm.

[0006] In the existing technology, the data of each tower are independent of each other, and when training the classifier, a large number of training sets are usually required. It is impossible to have enough training sets for each individual tower, especially for some training sets with danger labels, because in most cases there are not many dangerous situations for each tower, resulting in a lack of training sets.

[0007] Therefore, it is hoped that there will be a technical solution to solve or at least alleviate the above-mentioned deficiencies in the prior art. Summary of the Invention

[0008] The object of the present invention is to provide a multimodal data analysis method based on federated learning to overcome or at least alleviate at least one of the above-mentioned defects of the prior art.

[0009] One aspect of the present invention provides a multimodal data analysis method based on federated learning for monitoring a communication tower system. The communication tower system includes at least two communication towers and a central cloud server. Each communication tower is provided with an edge server, at least one camera device, and at least one sound pickup device. The edge server is connected to each camera device and each sound pickup device provided on the same communication tower. The multimodal data analysis method based on federated learning includes:

[0010] Perform the following operations for each edge server:

[0011] The edge server obtains the environmental images captured by each camera device connected to it and the audio information transmitted by each sound pickup device;

[0012] The edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server;

[0013] The edge server obtains the movement trajectory of the suspicious moving object based on the environmental images captured by each camera device;

[0014] The edge server extracts the trajectory features of the movement trajectory of the suspicious object;

[0015] The edge server obtains the audio information transmitted by each of the sound pickup devices during the time period when the movement trajectory of the suspicious object is captured;

[0016] The edge server extracts audio features from the audio information;

[0017] The edge server fuses the trajectory feature and the audio feature to obtain a fused feature;

[0018] The edge server inputs the fusion feature into the trained motion trajectory classifier based on federated learning, thereby obtaining a classification label, wherein the classification label includes a danger classification label;

[0019] When the classification label is a dangerous classification label, the edge server generates an alarm signal.

[0020] Optionally, before the edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server, the multimodal data analysis method based on federated learning further includes:

[0021] Train a motion trajectory classifier based on federated learning.

[0022] Optionally, the edge server obtaining a motion trajectory classifier based on federated learning that is trained by a central cloud server includes:

[0023] The central cloud server obtains the parameter information transmitted by each edge server respectively;

[0024] The central cloud server trains the global model based on the parameter information transmitted by each edge server, thereby obtaining feedback parameter information trained by the central server;

[0025] The central server transmits the feedback parameter information to each edge server;

[0026] Each edge server updates its own motion trajectory classifier through the obtained feedback parameter information, thereby obtaining a motion trajectory classifier based on federated learning trained by the central cloud server.

[0027] Optionally, the edge server obtaining a motion trajectory classifier based on federated learning trained by the central cloud server further includes:

[0028] Acquire a communication tower terrain database, where the communication tower terrain database includes at least two communication towers and preset pedestrian routes, and one communication tower corresponds to one or more preset pedestrian routes;

[0029] Obtaining each preset pedestrian route of each communication tower to form a trajectory route set;

[0030] Clustering the trajectory route set to obtain a clustering result, wherein the clustering result includes at least one cluster cluster;

[0031] The number of motion trajectory classifiers based on federated learning trained by the central cloud server is the same as the number of clusters, and one motion trajectory classifier based on federated learning trained by the central cloud server is used for one cluster;

[0032] The edge server inputting the fusion feature into the trained motion trajectory classifier based on federated learning includes:

[0033] The edge server obtains the movement trajectory of the suspicious object extracted by itself;

[0034] The edge server obtains a preset pedestrian route used by the suspicious object according to the movement trajectory of the suspicious object;

[0035] The edge server determines the cluster to which the preset pedestrian route used by the suspicious object belongs based on the preset pedestrian route used by the suspicious object;

[0036] The edge server obtains the motion trajectory classifier based on federated learning that is trained by the central cloud server and corresponds to the cluster to which it belongs;

[0037] The edge server inputs the fusion feature into the motion trajectory classifier based on federated learning that is trained by the central cloud server and corresponds to the cluster to which it belongs.

[0038] Optionally, the communication tower includes a plurality of distance sensors and an alarm device, and the multimodal data analysis method based on federated learning further includes:

[0039] After receiving the alarm signal, the edge server obtains the distance information of each distance sensor;

[0040] Determine whether the distance information transmitted by at least one distance sensor continues to decrease within a preset time range. If so,

[0041] The alarm signal is transmitted to the alarm device.

[0042] Optionally, before the edge server generates an alarm signal, the multimodal data analysis method based on federated learning further includes:

[0043] The edge server determines whether the moving target is a human according to each environment image, and if so,

[0044] The edge server generates an alarm signal.

[0045] Optionally, the edge server determining whether the moving target is a human according to each environment image includes:

[0046] The edge server obtains the federated learning-based image classifier trained by the central cloud server;

[0047] The edge server extracts image features of moving targets in the environment image;

[0048] The image features of the moving target are input into the image classifier based on federated learning trained by the central cloud server, thereby obtaining the classification label output by the image classifier based on federated learning trained by the central cloud server, wherein the classification label includes a human classification label. When the classification label is a human classification label,

[0049] Judged as human.

[0050] Optionally, the edge server extracting the trajectory features of the suspicious object's motion trajectory includes:

[0051] The edge server obtains the position of the moving target in each environment image;

[0052] The edge server obtains an environmental image in which the position of the moving target in the image satisfies a preset position condition;

[0053] The edge server extracts the moving target feature in the environment image where the position of the moving target in the image meets the preset position condition.

[0054] The present application also provides a communication tower system, which includes at least two communication towers and a central cloud server, each communication tower is provided with an edge server, at least one camera device and at least one sound pickup device, the edge server is connected to each camera device provided on the same communication tower and is connected to each sound pickup device provided on the same communication tower; wherein the edge server and the central cloud server cooperate to implement the multimodal data analysis method based on federated learning as described above.

[0055] Beneficial effects

[0056] This application has the following advantages:

[0057] This application forms an integrated network of each communication tower and the central cloud server, so that the motion trajectory classifier can be learned through federated learning without aggregating the data sets of each communication tower to the central cloud server, thereby achieving accurate classification of the classifier when each communication tower has a small data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flowchart of a multimodal data analysis method based on federated learning according to the first embodiment of the present application;

[0059] Figure 2 An electronic device used to implement Figure 1 The multimodal data analysis method based on federated learning shown;

[0060] Figure 3 yes Figure 1 Schematic diagram of preset pedestrian routes of communication towers in the multimodal data analysis method based on federated learning. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the implementation of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below in conjunction with the drawings in the embodiments of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be understood as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0062] Figure 1 This is a flowchart of the multimodal data analysis method based on federated learning in the first embodiment of the present application.

[0063] The multimodal data analysis method based on federated learning of the present application is used for monitoring a communication tower system. The communication tower system includes at least two communication towers and a central cloud server. Each communication tower is provided with an edge server, at least one camera device and at least one sound pickup device. The edge server is connected to each camera device provided on the same communication tower and is connected to each sound pickup device provided on the same communication tower.

[0064] It can be understood that there are multiple communication towers, each of which has an edge server, and each edge server transmits information to and from the central cloud server. Each communication tower has at least one camera device and at least one sound pickup device, and the edge server on each communication tower is connected to the camera device on the communication tower for transmitting data to each other.

[0065] In this embodiment, each communication tower may have multiple entrances due to its different locations, and therefore, multiple preset pedestrian routes. Figure 3 For example, Figure 3 The communication tower shown includes at least two preset pedestrian routes, namely the routes indicated by the two black lines in the figure. The two videos respectively indicate that people can travel to the tower location from these two directions.

[0066] like Figure 1 The multimodal data analysis methods based on federated learning shown include:

[0067] Perform the following operations for each edge server:

[0068] Step 1: The edge server obtains the environmental images captured by each camera device connected to it and the audio information transmitted by each sound pickup device;

[0069] Step 2: The edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server;

[0070] Step 3: The edge server obtains the movement trajectory of the suspicious moving object based on the environmental images captured by each camera device;

[0071] Step 4: The edge server extracts the trajectory features of the suspicious object's motion trajectory;

[0072] Step 5: The edge server obtains the audio information transmitted by each of the sound pickup devices during the time period when the movement trajectory of the suspicious object is captured;

[0073] Step 6: The edge server extracts audio features from the audio information;

[0074] Step 7: The edge server fuses the trajectory features and the audio features to obtain fused features;

[0075] Step 8: The edge server inputs the fused features into a trained motion trajectory classifier based on federated learning to obtain classification labels, including dangerous classification labels;

[0076] Step 9: When the classification label is a dangerous classification label, the edge server generates an alarm signal.

[0077] This application forms an integrated network of each communication tower and the central cloud server, so that the motion trajectory classifier can be learned through federated learning without aggregating the data sets of each communication tower to the central cloud server, thereby achieving accurate classification of the classifier when each communication tower has a small data set.

[0078] For example, a communication tower is a relatively dangerous device. If some children approach the communication tower out of curiosity, it may be extremely dangerous, especially if the children climb the tower. Therefore, it is necessary to predict the movement trajectory of suspicious objects (such as people) to determine whether the person is approaching the tower to a certain preset range. At this time, the movement trajectory can be used to predict. If it is predicted that the person is approaching, an alarm will be issued to prevent accidents.

[0079] In addition, the present application adopts an analysis method of multimodal data fusion. Multimodality generally includes front-end fusion (data level fusion), back-end fusion (decision level fusion) and intermediate fusion, etc. In this embodiment, the front-end fusion method is adopted.

[0080] This application considers image recognition on the one hand and audio data on the other hand (audio data mainly judges the approach or distance of an object by the size characteristics of the sound). Therefore, by fusing image features with audio features, the movement trajectory of suspicious objects can be identified more accurately.

[0081] In this embodiment, the principal component analysis method is used to reduce the dimensionality of the original data (image data and audio data).

[0082] In this embodiment, before the edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server, the multimodal data analysis method based on federated learning further includes:

[0083] Train a motion trajectory classifier based on federated learning.

[0084] In this embodiment, the edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server, including:

[0085] The central cloud server obtains the parameter information transmitted by each edge server respectively;

[0086] The central cloud server trains the global model based on the parameter information transmitted by each edge server, thereby obtaining feedback parameter information trained by the central server;

[0087] The central server transmits the feedback parameter information to each edge server;

[0088] Each edge server updates its own motion trajectory classifier through the obtained feedback parameter information, thereby obtaining a motion trajectory classifier based on federated learning trained by the central cloud server.

[0089] The method of this application has the following advantages:

[0090] By using distributed computing, the computing pressure will be distributed, and there will not be a prominent computing peak period. In addition, there will not be people on each communication tower most of the time. For example, there are relatively few people in remote areas at night, so this opportunity can be used to train neural networks (classifiers).

[0091] In addition, the method of the present application avoids the island effect and can connect the communication towers in a certain area, a certain city or even the whole country into a network for training, which greatly improves the data set and makes the prediction more accurate.

[0092] Finally, the method of the present application can also avoid data incommunicability problems caused by different data formats or confidentiality requirements between communication towers belonging to different companies.

[0093] In this embodiment, the edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server further includes:

[0094] Acquire a communication tower terrain database, where the communication tower terrain database includes at least two communication towers and preset pedestrian routes, and one communication tower corresponds to one or more preset pedestrian routes;

[0095] Obtaining each preset pedestrian route of each communication tower to form a trajectory route set;

[0096] Clustering the trajectory route set to obtain a clustering result, wherein the clustering result includes at least one cluster cluster;

[0097] The number of motion trajectory classifiers based on federated learning trained by the central cloud server is the same as the number of clusters, and one motion trajectory classifier based on federated learning trained by the central cloud server is used for one cluster;

[0098] The edge server inputting the fusion feature into the trained motion trajectory classifier based on federated learning includes:

[0099] The edge server obtains the movement trajectory of the suspicious object extracted by itself;

[0100] The edge server obtains a preset pedestrian route used by the suspicious object according to the movement trajectory of the suspicious object;

[0101] The edge server determines the cluster to which the preset pedestrian route used by the suspicious object belongs based on the preset pedestrian route used by the suspicious object;

[0102] The edge server obtains the motion trajectory classifier based on federated learning that is trained by the central cloud server and corresponds to the cluster to which it belongs;

[0103] The edge server inputs the fusion feature into the motion trajectory classifier based on federated learning that is trained by the central cloud server and corresponds to the cluster to which it belongs.

[0104] As Figure 3 As shown in the figure, each communication tower has a different location, which will lead to different pedestrian routes. Therefore, it is necessary to cluster the pedestrian routes, so as to bring similar pedestrian routes together and train similar pedestrian routes with a unified classifier. In this way, the pedestrian routes are used as the criterion instead of the individual communication towers. By comparing similar pedestrian routes to ensure the convergence of the model, each classifier can be made more accurate and targeted.

[0105] For example, there are two communication towers, one of which is Figure 3 As shown, the communication tower has two pedestrian routes (A and B), while the other communication tower has two pedestrian routes (C and D). Similar routes are obtained by clustering (it can be understood that clustering is a more appropriate method when the data set is relatively large) or by directly comparing the similarity of the two routes (when the data set is relatively small, direct fitting is also possible), and similar routes are formed into a set, thereby generating a motion trajectory classifier for the set.

[0106] During training, the training set of similar routes is also used in the training of the motion trajectory classifier generated by the set, so as to make the training more accurate.

[0107] In this embodiment, the communication tower includes multiple distance sensors and alarm devices, and the multimodal data analysis method based on federated learning further includes:

[0108] After receiving the alarm signal, the edge server obtains the distance information of each distance sensor;

[0109] Determine whether the distance information transmitted by at least one distance sensor continues to decrease within a preset time range. If so,

[0110] The alarm signal is transmitted to the alarm device.

[0111] In this embodiment, before the edge server generates an alarm signal, the multimodal data analysis method based on federated learning further includes:

[0112] The edge server determines whether the moving target is a human based on each environment image. If so,

[0113] The edge server generates an alarm signal.

[0114] In this embodiment, the edge server determines whether the moving target is a human according to each environment image, including:

[0115] The edge server obtains the federated learning-based image classifier trained by the central cloud server;

[0116] The edge server extracts image features of moving targets in the environment image;

[0117] The image features of the moving target are input into the image classifier based on federated learning trained by the central cloud server, thereby obtaining the classification label output by the image classifier based on federated learning trained by the central cloud server, wherein the classification label includes a human classification label. When the classification label is a human classification label,

[0118] Judged as human.

[0119] It is understandable that determining whether a person is human through images is a prior art and will not be elaborated here.

[0120] It is understandable that the image classifier used to determine whether an image is human also adopts a training method based on federal learning.

[0121] In this embodiment, the labels used in the training set can be self-contained or manually annotated by the user using the Labelimg annotation tool, and the data in the training set can also be data obtained by the user through actual walking. For example, Figure 3 The route shown is walked 1000 times, and the training set is obtained by varying the time, the walking time, and the route.

[0122] In this embodiment, the behavior of a suspicious object can be estimated by a classifier based on its motion trajectory. For example, if the motion trajectory of the suspicious object is a relatively straight and uniform motion trajectory along a preset pedestrian route, then the pedestrian's purpose is likely to be passing by. On the contrary, if the pedestrian's motion trajectory is relatively slow and tortuous, or if the pedestrian often deviates from and returns to the preset pedestrian route, then the pedestrian is likely to be a tourist or a child playing around. In this case, a warning is required, especially when the pedestrian is getting closer and closer to the communication tower on the motion trajectory.

[0123] In this embodiment, the environmental image obtained by this application is an environmental image of a certain time period, for example, an environmental image within 300 seconds. On the one hand, only enough time can obtain a relatively complete movement trajectory of the suspicious object. On the other hand, the purpose reflected by the movement trajectory obtained over a longer period of time is clearer. If the suspicious object has been moving around the communication tower within 300 seconds and has no intention of leaving, it is very likely that an alarm will be required to drive it away.

[0124] In this embodiment, by setting preset pedestrian routes, an initial clustering condition can be provided. This is because a communication tower may have multiple approach routes, each with a different walking direction. Without clustering and classification, the discrete amount is relatively large, making convergence difficult. The method of this application can be used to accelerate the training process by clustering routes. In addition, in many cases, parts of the communication tower have guardrails or buildings, where normal pedestrians cannot walk. Therefore, these areas can be disregarded. If there are traces of objects moving in these areas, they are likely not humans, but rather animals or other objects, and therefore do not need to be considered.

[0125] In this embodiment, the preset pedestrian route refers to a normal route that a person can walk in a scene captured within the visual range that can be captured by the camera device in theory.

[0126] In this embodiment, various possible routes can be manually set in the training set of this application to enrich the training set.

[0127] In this embodiment, the edge server extracts the trajectory features of the suspicious object's motion trajectory including:

[0128] The edge server obtains the position of the moving target in each environment image;

[0129] The edge server obtains an environmental image in which the position of the moving target in the image meets a preset position condition;

[0130] The edge server extracts the moving target feature in the environment image where the position of the moving target in the image meets the preset position condition.

[0131] In this embodiment, the motion trajectory is obtained in the following manner:

[0132] For example, there are 100 environmental images. These 100 environmental images are obtained from continuous frames and arranged in sequence from 1 to 100, representing continuous images within 100 seconds (assuming 1 frame per second). By understanding the position of the moving target in each frame, the movement trajectory of the moving target can be understood.

[0133] In this embodiment, the present application can further prevent misjudgment by adding a distance sensor.

[0134] In this embodiment, the edge server determines whether the moving target is a human according to each environment image, including:

[0135] The edge server obtains the trained human classifier;

[0136] The edge server extracts the moving target features of the environment image;

[0137] The moving target features are input into the human classifier to determine whether it is a human.

[0138] In this embodiment, the edge server extracts the moving target features of the environment image including:

[0139] The edge server obtains the position of the moving target in each environment image;

[0140] The edge server obtains an environmental image in which the position of the moving target in the image meets a preset position condition;

[0141] The edge server extracts the moving target feature in the environment image where the position of the moving target in the image meets the preset position condition.

[0142] In this embodiment, there may be hundreds or thousands of photos that can capture moving targets. If each photo is identified, a large amount of computing power will be wasted. Therefore, only selecting moving targets that meet the preset position conditions can save computing power.

[0143] In this embodiment, the preset position condition is close to the central area of ​​the image, so that the overall characteristics of the motion trajectory can be obtained most completely.

[0144] The present application also provides a communication tower system, which includes at least two communication towers and a central cloud server, each communication tower is provided with an edge server, at least one camera device and at least one sound pickup device, the edge server is connected to each camera device provided on the same communication tower and is connected to each sound pickup device provided on the same communication tower; wherein the edge server and the central cloud server cooperate to implement the multimodal data analysis method based on federated learning as described above.

[0145] It should be noted that the aforementioned explanation of the method embodiment is also applicable to the system of this embodiment and will not be repeated here.

[0146] The present application also provides an electronic device. In this embodiment, the electronic device is an edge server, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the above-mentioned multimodal data analysis method based on federated learning is implemented.

[0147] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned multimodal data analysis method based on federated learning.

[0148] Figure 2 This is an exemplary structural diagram of an electronic device that can implement the multimodal data analysis method based on federated learning provided according to an embodiment of the present application.

[0149] like Figure 2As shown, the electronic device includes an input device 501, an input interface 502, a central processing unit 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processing unit 503, the memory 504, and the output interface 505 are interconnected via a bus 507. The input device 501 and the output device 506 are connected to the bus 507 via the input interface 502 and the output interface 505, respectively, and are then connected to other components of the electronic device. Specifically, the input device 504 receives input information from the outside and transmits the input information to the central processing unit 503 via the input interface 502; the central processing unit 503 processes the input information based on the computer-executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 via the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.

[0150] That is to say, Figure 2 The electronic device shown may also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination of the computer executable instructions when executing the computer executable instructions. Figure 1 Described is a multimodal data analysis method based on federated learning.

[0151] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504, configured to store executable program code; one or more processors 503, configured to run the executable program code stored in the memory 504 to execute the multimodal data analysis method based on federated learning in the above embodiment.

[0152] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0153] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0154] Computer-readable media include permanent and non-permanent, removable and non-removable media, and media can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] Furthermore, it is clear that the word "comprising" does not exclude other units or steps. Multiple units, modules, or devices recited in a device claim may also be implemented by a single unit or overall device through software or hardware. Terms such as first and second are used to identify names, not to indicate any particular order.

[0157] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and a part of the module, program segment or code includes one or more executable instructions for realizing the prescribed logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or overall flow chart can be implemented using a dedicated hardware-based system that performs the prescribed function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0158] The processor referred to in this embodiment may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0159] The memory can be used to store computer programs and / or modules. The processor implements various functions of the device / terminal equipment by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0160] In this embodiment, if the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0161] It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. Although the present application is disclosed above with reference to preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0162] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made based on the present invention. Therefore, such modifications and improvements, which do not depart from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A multimodal data analysis method based on federated learning, for use in monitoring a communication tower system, wherein the communication tower system comprises at least two communication towers and a central cloud server, wherein each communication tower is provided with an edge server, at least one camera device, and at least one sound pickup device, wherein the edge server is connected to each camera device and each sound pickup device provided on the same communication tower, characterized in that: The multimodal data analysis method based on federated learning includes: Perform the following operations for each edge server: The edge server obtains the environmental images captured by each camera device connected to it and the audio information transmitted by each sound pickup device; The edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server; The edge server obtains the movement trajectory of the suspicious moving object based on the environmental images captured by each camera device; The edge server extracts the trajectory features of the movement trajectory of the suspicious moving object; The edge server obtains the audio information transmitted by each of the sound pickup devices during the time period when the movement trajectory of the suspicious moving object is captured; The edge server extracts audio features from the audio information; The edge server fuses the trajectory feature and the audio feature to obtain a fused feature; The edge server inputs the fusion feature into the trained motion trajectory classifier based on federated learning, thereby obtaining a classification label, wherein the classification label includes a danger classification label; When the classification label is a dangerous classification label, the edge server generates an alarm signal; Before the edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server, the multimodal data analysis method based on federated learning further includes: Train the motion trajectory classifier based on federated learning; The edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server, including: The central cloud server obtains the parameter information transmitted by each edge server respectively; The central cloud server trains the global model based on the parameter information transmitted by each edge server, thereby obtaining feedback parameter information trained by the central server; The central server transmits the feedback parameter information to each edge server; Each edge server updates its own motion trajectory classifier using the feedback parameter information it obtains, thereby obtaining a motion trajectory classifier based on federated learning trained by the central cloud server. The edge server obtains the motion trajectory classifier based on federated learning trained by the central cloud server further comprising: Acquire a communication tower terrain database, where the communication tower terrain database includes at least two communication towers and preset pedestrian routes, and one communication tower corresponds to one or more preset pedestrian routes; Obtaining each preset pedestrian route of each communication tower to form a trajectory route set; Clustering the trajectory route set to obtain a clustering result, wherein the clustering result includes at least one cluster cluster; The number of motion trajectory classifiers based on federated learning trained by the central cloud server is the same as the number of clusters, and one motion trajectory classifier based on federated learning trained by the central cloud server is used for one cluster; The edge server inputting the fusion feature into the trained motion trajectory classifier based on federated learning includes: The edge server obtains the motion trajectory of the suspicious moving object extracted by itself; The edge server obtains a preset pedestrian route used by the suspicious moving object according to the movement trajectory of the suspicious moving object; The edge server determines the cluster to which the preset pedestrian route used by the suspicious moving object belongs based on the preset pedestrian route used by the suspicious moving object; The edge server obtains the motion trajectory classifier based on federated learning that is trained by the central cloud server and corresponds to the cluster to which it belongs; The edge server inputs the fusion feature into the motion trajectory classifier based on federated learning that is trained by the central cloud server and corresponds to the cluster to which it belongs.

2. The multimodal data analysis method based on federated learning according to claim 1, wherein the communication tower includes a plurality of distance sensors and an alarm device, wherein: The multimodal data analysis method based on federated learning further includes: After receiving the alarm signal, the edge server obtains the distance information of each distance sensor; Determine whether the distance information transmitted by at least one distance sensor continues to decrease within a preset time range. If so, The alarm signal is transmitted to the alarm device.

3. The multimodal data analysis method based on federated learning according to claim 2, characterized in that: Before the edge server generates an alarm signal, the multimodal data analysis method based on federated learning further includes: The edge server determines whether the moving target is a human according to each environment image, and if so, The edge server generates an alarm signal.

4. The multimodal data analysis method based on federated learning according to claim 3, characterized in that: The edge server determines whether the moving target is a human according to each environment image, including: The edge server obtains the federated learning-based image classifier trained by the central cloud server; The edge server extracts image features of moving targets in the environment image; The image features of the moving target are input into the image classifier based on federated learning trained by the central cloud server, thereby obtaining the classification label output by the image classifier based on federated learning trained by the central cloud server, wherein the classification label includes a human classification label. When the classification label is a human classification label, Judged as human.

5. The multimodal data analysis method based on federated learning according to claim 4, characterized in that: The edge server extracts the trajectory features of the suspicious moving object's trajectory including: The edge server obtains the position of the moving target in each environment image; The edge server obtains an environmental image in which the position of the moving target in the image satisfies a preset position condition; The edge server extracts the moving target feature in the environment image where the position of the moving target in the image meets the preset position condition.

6. A communication tower system, characterized in that: The communication tower system includes at least two communication towers and a central cloud server, each communication tower is provided with an edge server, at least one camera device and at least one sound pickup device, the edge server is connected to each camera device provided on the same communication tower and is connected to each sound pickup device provided on the same communication tower; wherein the edge server and the central cloud server cooperate to implement the multimodal data analysis method based on federated learning as described in any one of claims 1 to 5.

Images