Object recognition method, electronic device, storage medium and program product

Abstract

This application provides an object recognition method that can be applied to the field of artificial intelligence technology. The method includes: obtaining multiple time series signals of different modalities related to a target area collected by multiple types of sensors; performing time series feature extraction on each of the multiple time series signals to obtain multiple time series features; reconstructing the multiple time series features to obtain reconstructed features; extracting semantic features from the reconstructed features to obtain semantic features; post-processing the semantic features to obtain target features; and using the target features to perform object recognition on the target area to obtain recognition results. This application also provides an electronic device, a storage medium, and a program product.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, specifically to multimodal signal processing technology, and more specifically to an object recognition method, electronic device, storage medium, and program product. Background Art

[0002] Computer vision is a key branch of artificial intelligence that focuses on converting information in images or videos into a form that computers can understand and process. Object recognition is a crucial task within computer vision, aiming to identify objects in images or videos and classify them as specific categories.

[0003] In actual applications, there are still challenges in recognizing some complex scenes such as dynamic objects. Summary of the Invention

[0004] In view of the above problems, the present application provides an object recognition method, a server, a storage medium, and a program product for improving the effectiveness of object recognition in complex scenes.

[0005] According to a first aspect of the present application, there is provided an object recognition method, comprising: acquiring multiple time series signals of different modalities about a target area collected by multiple types of sensors; performing time series feature extraction on the multiple time series signals to obtain multiple time series features; performing feature reconstruction on the multiple time series features to obtain reconstructed features; performing semantic feature extraction on the reconstructed features to obtain semantic features; post-processing the semantic features to obtain target features; and performing object recognition on the target area using the target features to obtain a recognition result.

[0006] The second aspect of the present application provides an object recognition device, including: a first module for acquiring multiple time series signals of different modalities about a target area collected by multiple types of sensors; a second module for performing time series feature extraction on the multiple time series signals to obtain multiple time series features; a third module for performing feature reconstruction on the multiple time series features to obtain reconstructed features; a fourth module for performing semantic feature extraction on the reconstructed features to obtain semantic features; a fifth module for post-processing the semantic features to obtain target features; and a sixth module for performing object recognition on the target area using the target features to obtain a recognition result.

[0007] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0008] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0009] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0011] Figure 1 An application scenario diagram of an object recognition method according to an embodiment of the present application is shown;

[0012] Figure 2 A flow chart of an object recognition method according to an embodiment of the present application is shown;

[0013] Figure 3 shows a model structure diagram of an object recognition model according to an embodiment of the present application;

[0014] Figure 4A shows a network structure diagram of a reconstruction module according to an embodiment of the present application;

[0015] Figure 4B shows a network structure diagram of a reconstruction module according to another embodiment of the present application;

[0016] Figure 5 The following is a flowchart of an unsupervised training of a sparse autoencoder according to an embodiment of the present application;

[0017] Figure 6 shows a network structure diagram of an object recognition model according to another embodiment of the present application;

[0018] Figure 7 A block diagram of an electronic device suitable for implementing an object recognition method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0021] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0022] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] In object recognition, images or videos can be used as the processed signal to identify the target area. However, this method only uses a single piece of information. Furthermore, the processing method is generally image processing technology, which is also relatively simple.

[0024] Therefore, in complex scenes, the effectiveness of object recognition is difficult to guarantee.

[0025] An embodiment of the present application provides an object recognition method, including: acquiring multiple time series signals of different modalities about a target area collected by multiple types of sensors; performing time series feature extraction on the multiple time series signals respectively to obtain multiple time series features; performing feature reconstruction on the multiple time series features to obtain reconstructed features; performing semantic feature extraction on the reconstructed features to obtain semantic features; post-processing the semantic features to obtain target features; and performing object recognition on the target area using the target features to obtain a recognition result.

[0026] The object recognition method provided in the embodiment of the present application is used to process multiple time series signals of different modalities. The processing methods include feature reconstruction, semantic feature extraction, and post-processing. Feature reconstruction can be used to associate time series features of multiple time series signals, and semantic feature extraction methods can be used to extract high-level semantic features, thereby deepening the analysis and improving the effectiveness, breadth, and depth of the signals to be analyzed, thereby improving the effectiveness of object recognition in complex scenes.

[0027] Figure 1 The following diagram shows an application scenario of the object recognition method according to an embodiment of the present application.

[0028] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first sensor 101, a second sensor 102, a third sensor 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first sensor 101, the second sensor 102, the third sensor 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0029] The user may use the first sensor 101 , the second sensor 102 , and the third sensor 103 to interact with the server 105 via the network 104 to send signals, etc.

[0030] The first sensor 101 , the second sensor 102 , and the third sensor 103 may be different types of sensors, including cameras, radars, audio collectors, etc., but are not limited thereto, and may also include environmental sensors such as temperature, humidity, and light.

[0031] The server 105 may be a server that provides various services, such as a background management server that processes signals collected by the user using the first sensor 101 , the second sensor 102 , and the third sensor 103 (only as an example).

[0032] It should be noted that the object recognition method provided in the embodiment of the present application can generally be executed by the server 105. Accordingly, the object recognition device provided in the embodiment of the present application can generally be set in the server 105. However, it is not limited to this. The object recognition method provided in the embodiment of the present application can also be executed by a terminal device. Accordingly, the object recognition device provided in the embodiment of the present application can generally be set in a terminal device. Optionally, the terminal device can be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, etc.

[0033] It should be understood that Figure 1The number of the first sensor, the second sensor, the third sensor, the network and the server in the embodiment is only illustrative. According to the implementation requirements, there can be any number of the first sensor, the second sensor, the third sensor, the network and the server.

[0034] The following will be based on Figure 1 The scene described by Figures 2 to 6 The object recognition method of the disclosed embodiment is described in detail.

[0035] Figure 2 A flow chart of an object recognition method according to an embodiment of the present application is shown.

[0036] like Figure 2 As shown, the object recognition method of this embodiment includes operations S210 to S260.

[0037] In operation S210 , a plurality of time series signals of different modalities related to a target area collected by a plurality of types of sensors are acquired.

[0038] In operation S220 , time series features are extracted from the multiple time series signals to obtain multiple time series features.

[0039] In operation S230 , feature reconstruction is performed on the plurality of time series features to obtain reconstructed features.

[0040] In operation S240 , semantic features are extracted from the reconstructed features to obtain semantic features.

[0041] In operation S250 , the semantic features are post-processed to obtain target features.

[0042] In operation S260 , object recognition is performed on the target area using the target features to obtain a recognition result.

[0043] Mode can be understood as the type of signal, but is not limited to this. It can also be understood as the source type of the signal.

[0044] Multiple signals of different modalities are referred to as multimodal signals. These signals can include, for example, text, images, audio, video, point clouds, and optical signals. These signals have different characteristics and representations. Correspondingly, multiple types of sensors can include cameras, audio collectors, radars, temperature sensors, pressure sensors, and speed sensors.

[0045] Multiple sensors of different types can be used to collect signals from the target area to describe the target area from multiple different dimensions, thereby obtaining multiple time series signals of different modalities related to the target area.

[0046] A timing signal can be understood as a sequence of signals arranged in chronological order.

[0047] Time series feature extraction can be understood as extracting features from time series signals to extract features related to time series knowledge. For example, features representing the changing trends of an object's movements at different moments can be extracted. Models such as recurrent neural networks (RNNs) or their variants, and long short-term memory networks (LSTMs) can be used to extract features from time series signals to obtain time series features related to time series knowledge.

[0048] Feature reconstruction can be understood as transforming features, including at least one of the following: sparsification, transposition, concatenation of multiple time series features, and dimensionality reduction. Feature reconstruction can improve the coupling between multiple time series features that represent object characteristics from different dimensions.

[0049] Optionally, the feature reconstruction provided in the embodiments of the present application may include sparse reconstruction, thereby addressing the high data volume of multiple time series features, while highlighting effective and important features and reducing the amount of data processing.

[0050] Semantic feature extraction can be understood as in-depth feature extraction to obtain high-level semantic features. Features can be processed using convolutional neural networks (CNNs), attention mechanisms, encoder-decoders (Transformers), and other techniques.

[0051] Post-processing can be understood as reprocessing of features, such as normalization, pooling, activation, etc.

[0052] Object recognition can be understood as classification. Objects in a target area can be classified, thereby obtaining a recognition result that characterizes the object's category. For example, a recognition result can characterize whether the object in the target area is normal. Another example is a recognition result that characterizes the specific characteristic category of the object in the target area.

[0053] The object recognition method provided in the embodiment of the present application is used to process multiple time series signals of different modalities. The processing methods include feature reconstruction, semantic feature extraction, and post-processing. Feature reconstruction can be used to associate time series features of multiple time series signals, and semantic feature extraction methods can be used to extract high-level semantic features, thereby deepening the analysis and improving the effectiveness, breadth, and depth of the signals to be analyzed, thereby improving the effectiveness of object recognition in complex scenes.

[0054] The above describes the overall process of object recognition. Optionally, the object recognition method can be performed using an object recognition model. The following describes each operation in detail in conjunction with the object recognition model.

[0055] Figure 3 A model structure diagram of an object recognition model according to an embodiment of the present application is shown.

[0056] like Figure 3 As shown, the object recognition model M300 may sequentially include a pre-processing module M310, a temporal feature extraction module M320, a reconstruction module M330, a semantic feature extraction module M340, a post-processing module M350, and a recognition module M360.

[0057] The preprocessing module M310 is used to acquire multiple time series signals of different modalities related to the target area, collected by multiple types of sensors. The time series feature extraction module M320 is used to extract time series features from the multiple time series signals to obtain multiple time series features. The reconstruction module M330 is used to reconstruct the multiple time series features to obtain reconstructed features. The semantic feature extraction module M340 is used to extract semantic features from the reconstructed features to obtain semantic features. The post-processing module M350 is used to post-process the semantic features to obtain target features. The recognition module M360 is used to process the target features to perform object recognition on the target area to obtain recognition results.

[0058] The operations performed by each module in the object recognition model will be described in detail below.

[0059] According to the embodiments of the present application, Figure 2 Operation S210, in which multiple time series signals of different modalities related to a target area are acquired by different sensors, may include acquiring multiple initial time series signals of different modalities related to the target area acquired by multiple types of sensors. Object detection is performed on the video signals in the multiple initial time series signals to obtain an object detection result. If the object detection result indicates that a target object is present in the target area within a target time period, signal extraction is performed on each of the multiple initial time series signals based on the target time period to obtain multiple time series signals.

[0060] The multiple initial time-series signals may be signals within a preset time period acquired by a sensor. Target detection may be performed on video signals within the multiple initial time-series signals to obtain a target detection result. For example, target detection may be performed on video signals acquired by a camera within a period of two hours to obtain a target detection result.

[0061] There are no specific restrictions on the object detection algorithm; for example, it may include R-CNN (Region-based Convolutional Neural Networks), Fast R-CNN, and SPP (Spatial Pyramid Pooling). The video signal can be processed by frame extraction to obtain multiple video frames. The video frames are input into the object detection unit included in the preprocessing module, where object detection is performed on the video frames using the object detection algorithm to obtain a detection result indicating the presence of the target object. The object detection result can be obtained based on the time sequence detection results that match the video signal. For example, if 200 video frames are extracted within 2 hours, the object detection result can be obtained based on the 200 detection results corresponding to these 200 video frames. Each detection result can be timestamped to determine the time period during which the target object was present.

[0062] This target detection result can indicate whether the target object exists in the target area during the target time period. For example, if the doorway is the target area and the video signal is detected between 9:00 PM and 10:30 PM, it can be determined that no target object appears in the target area. Therefore, subsequent object recognition operations are no longer necessary during this time period, and processing of the multiple initial time series signals for this time period can be stopped.

[0063] When the target detection result indicates that a target object exists in the target area during a target period, for example, from 10:30 to 11:00, signal extraction can be performed on the multiple initial time series signals based on the target period to obtain multiple time series signals.

[0064] In this way, only the signals within the target time period where the target object exists can be extracted to obtain multiple time series signals. Compared with the method of performing object recognition on multiple initial time series signals, the operation of using multiple time series signals for object recognition can perform preliminary screening of multiple initial time series signals through object detection, thereby improving processing efficiency and processing accuracy.

[0065] According to an embodiment of the present application, extracting signals from multiple initial time series signals based on a target time period in a target detection result to obtain multiple time series signals may include: extracting signals from multiple initial time series signals based on the target time period to obtain multiple signal segments. Based on the sensor's signal acquisition frequency and the target signal frequency, extracting key signals from signal segments that match the sensor to obtain the time series signals. The target signal frequency is determined based on the signal acquisition frequencies of multiple types of sensors.

[0066] Using the video signal described above as an example, the initial time series signal may include video signals from 9:00 to 11:00. Object detection results indicate that the target object exists in the target area between a target time period, such as 10:30 to 11:00. Based on this target time period, the video signal can be intercepted to obtain a video signal segment between 10:30 and 11:00.

[0067] Optionally, the initial time series signals of other modes may also be intercepted by only intercepting the time series signals between 10:30 and 11:00 to obtain signal segments.

[0068] Multiple signal segments can be directly used as multiple time series signals. However, this is not a limitation. Considering the different signal acquisition frequencies of various types of sensors, key signals can also be extracted from the signal segments to obtain time series signals.

[0069] The sensor's signal acquisition frequency can be compared with the target signal frequency. When the signal acquisition frequency is greater than the target signal frequency, key signal extraction can be performed on the signal segment collected by the sensor to obtain a timing signal consistent with the target signal frequency.

[0070] The target signal frequency can be determined by selecting the lowest signal acquisition frequency among multiple sensor types. This allows for simple key signal extraction. However, this is not a limitation. Alternatively, the target signal frequency can be determined by selecting the highest signal acquisition frequency, requiring data supplementation for signal segments with frequencies below the target signal frequency. Alternatively, data supplementation can be performed using the average of two adjacent signals.

[0071] For example, the signal segment includes , you can use Find the mean and add to Get the timing signal .

[0072] This ensures that the time periods of the time series signals are the same, and the time corresponding to the signals collected in the same time period is the same, thereby improving the effectiveness and reference value of object recognition by associating multiple time series signals.

[0073] The above describes the preprocessing operations performed by the preprocessing module. The following describes the temporal feature extraction performed by the temporal feature extraction module.

[0074] According to the embodiments of the present application, Figure 2 Operation S220 shown, extracting timing features from multiple timing signals respectively to obtain multiple timing features, may include: using multiple gated recurrent units connected in parallel to extract timing features from timing signals matching the gated recurrent units to obtain multiple timing features.

[0075] Optionally, the time series feature extraction module may include multiple gated recurrent units (GRUs) connected in parallel, each of which processes a time series signal to obtain a time series feature.

[0076] Specifically, the GRU uses an update gate and a reset gate to determine which information ultimately serves as the GRU's output. This prevents the GRU from erasing historical information over time, retaining relevant information and passing it to the next moment. This allows the GRU to utilize all information in the time series signal and avoid the vanishing gradient problem.

[0077] Using multiple gated recurrent units in parallel to process multiple time series signals can improve processing efficiency. In addition, processing time series signals with a gated neural network of gated recurrent units can efficiently capture and fuse time series features.

[0078] The above section describes the temporal feature extraction module in detail. The following section describes the feature reconstruction module in detail.

[0079] According to the embodiments of the present application, Figure 2 Operation S230, which reconstructs multiple time series features to obtain reconstructed features, may include: concatenating the multiple time series features to obtain concatenated features; determining target model parameters from multiple model parameters that match the feature scale of the concatenated features; performing sparsification reconstruction on the concatenated features using a sparse autoencoder configured with the target model parameters to obtain initial reconstructed features; and performing feature extraction on the initial reconstructed features to obtain reconstructed features.

[0080] Multiple time series features can be concatenated along the channel dimension to obtain a concatenated feature. However, this is not a limitation. Alternatively, concatenation (a fusion function) can be used to obtain a concatenated feature. The characteristic scale of the concatenated feature is determined. The characteristic scale can be understood as the height, width, and depth of the feature. Multiple model parameters can be pre-built to match the characteristic scales of the various concatenated features, generating a mapping relationship between the characteristic scales and the model parameters. Based on this mapping relationship, target model parameters are determined to match the characteristic scale of the concatenated feature.

[0081] A sparse autoencoder (SAE), an unsupervised machine learning algorithm configured with the target model parameters, performs a sparse reconstruction of the concatenated features to obtain initial reconstructed features. Specifically, sparse reconstruction can include feature extraction and dimensionality reduction. By learning the sparse representation of the concatenated features, important features in the data are extracted. This ensures that key information is retained while reducing the data dimension.

[0082] Initial reconstructed features can be used as reconstructed features. However, this is not a limitation. Feature extraction can also be performed on the initial reconstructed features to obtain reconstructed features. The feature extraction method is not limited and can be performed using, for example, a convolutional neural network or a residual block.

[0083] Figure 4A A network structure diagram of a reconstruction module according to an embodiment of the present application is shown.

[0084] like Figure 4A As shown, the reconstruction module M330 may include a splicing unit M331, a first reconstruction unit M332, and a feature extraction unit M333.

[0085] like Figure 4A As shown, a splicing unit M331 can be used to splice multiple time series features to obtain spliced ​​features. Target model parameters that match the characteristic scale of the spliced ​​features are determined from multiple model parameters. A sparse autoencoder configured with the target model parameters is used as a first reconstruction unit. The spliced ​​features are sparsely reconstructed by the first reconstruction unit M332 to obtain initial reconstructed features. Feature extraction unit M333 is used to extract features from the initial reconstructed features to obtain reconstructed features.

[0086] By utilizing the feature reconstruction method provided in the embodiment of the present application, multiple time series features can be spliced ​​together, and an appropriate sparse reconstruction network can be selected based on the characteristic scale of the spliced ​​features, thereby improving processing flexibility, reducing the amount of subsequent data processing, and effectively highlighting important features.

[0087] According to an embodiment of the present application, the object recognition method may further include: repeatedly performing sparse reconstruction multiple times.

[0088] For example, the initial reconstructed features after feature extraction are used as the first reconstructed features. If the feature scale of the first reconstructed features is the same as that of the spliced ​​features, the first reconstructed features are sparsified and reconstructed using a sparse autoencoder configured with the target model parameters to obtain a second reconstructed feature. The first and second reconstructed features are fused to obtain the reconstructed features.

[0089] Figure 4B A network structure diagram of a reconstruction module according to another embodiment of the present application is shown.

[0090] like Figure 4B As shown, the reconstruction module M330 ′ may include a splicing unit M331 , a first reconstruction unit M332 , a feature extraction unit M333 , a second reconstruction unit M334 , and a fusion unit M335 .

[0091] like Figure 4B As shown, a splicing unit M331 can be used to splice multiple time series features to obtain a spliced ​​feature. A target model parameter that matches the feature scale of the spliced ​​feature is determined from multiple model parameters. A sparse autoencoder configured with the target model parameters is used as a first reconstruction unit, and the spliced ​​feature is sparsely reconstructed using the first reconstruction unit M332 to obtain an initial reconstructed feature. The initial reconstructed feature is extracted using the feature extraction unit M333 to obtain a first reconstructed feature. A sparse autoencoder configured with the target model parameters is used as a second reconstruction unit M334, and the first reconstructed feature is sparsely reconstructed using the second reconstruction unit M334 to obtain a second reconstructed feature. The first reconstructed feature and the second reconstructed feature are fused using the fusion unit M335 to obtain a reconstructed feature.

[0092] The first reconstructed feature and the second reconstructed feature can be fused by matrix multiplication, but the method is not limited thereto, and a channel dimension addition method can also be used, as long as the fusion can be performed.

[0093] The second reconstructed features can be used to further sparse the features and abstract more important features. The second reconstructed features can be fused with the first reconstructed features to retain the integrity of the original features while further highlighting the important features.

[0094] According to an embodiment of the present application, the object recognition method may further include the following operations.

[0095] For example, if the feature scales of the first reconstructed feature and the spliced ​​feature are different, the first reconstructed feature can be padded to obtain a preprocessed reconstructed feature. The preprocessed reconstructed feature is then sparsely reconstructed using a sparse autoencoder configured with the target model parameters to obtain a second reconstructed feature.

[0096] There is no limitation on the feature element padding process. For example, the weighted sum of adjacent elements may be used to pad the element at that position. However, this is not limited to this and may also include padding with "0". Any method can be used to make the feature scale of the first reconstructed feature and the spliced ​​feature the same.

[0097] For example, the “0” padding method can make the feature dimensions the same without adding extra noise information.

[0098] When the feature scales are different, the feature elements of the first reconstructed feature are padded, so that the target model parameters can be adapted to the preprocessed reconstructed feature to be processed, thereby improving the effect of subsequent sparse reconstruction.

[0099] According to another embodiment of the present application, the object recognition method may further include the following operations. For example, when the feature scales of the first reconstructed feature and the spliced ​​feature are different, second target model parameters that match the first reconstructed feature are determined from a plurality of model parameters. The preprocessed reconstructed feature is sparsified and reconstructed using a sparse autoencoder configured with the second target model parameters to obtain a second reconstructed feature.

[0100] When the feature scales are different, determining the second target model parameters that match the first reconstructed features from multiple model parameters can make full use of the existing information and avoid additional processing operations to fill in the feature elements. It can also make the second target model parameters adapt to the first reconstructed features, thereby improving the effect of subsequent sparse reconstruction.

[0101] According to an embodiment of the present application, the object recognition method may further include determining a plurality of model parameters.

[0102] Optionally, the multiple model parameters may be determined by using multiple sample features of different feature scales to train a sparse autoencoder configured with the initial model parameters to obtain the multiple model parameters.

[0103] Exemplarily, training a sparse autoencoder configured with initial model parameters using sample features may include the following operations: sparsifying and reconstructing the sample features using the sparse autoencoder configured with the initial model parameters to obtain sample reconstructed features; determining a reconstruction loss function based on the sample features and the sample reconstructed features; determining a regularization loss function based on the initial model parameters and a sample scale parameter representing a characteristic scale of the sample features; and adjusting the initial model parameters based on the reconstruction loss function and the regularization loss function to obtain model parameters that match the sample scale parameters.

[0104] Optionally, using a sparse autoencoder configured with initial model parameters to sparsely reconstruct the sample features to obtain the sample reconstructed features may include: encoding the sample features using the sparse autoencoder to obtain sample encoded features; fusing the sample encoded features with a weight matrix and then concatenating the features with a bias term to obtain sample sparse features; and activating the sample sparse features to obtain sample reconstructed features.

[0105] Sparse autoencoders can perform matrix multiplication on sample features using an encoding weight matrix to obtain sample encoded features. However, this is not limited to this approach. Alternatively, a network structure with a self-attention mechanism can be used to encode sample features based on the encoding weight matrix to obtain sample encoded features.

[0106] The sample encoding features are fused with a weight matrix, such as a decoding weight matrix, by performing matrix multiplication, and then combined with a bias term to obtain a sample sparsification feature. The sample sparsification feature is activated to obtain a sample reconstruction feature.

[0107] Optionally, the sparse autoencoder may perform the following operation of formula (1).

[0108] ;Formula (1)

[0109] in, represents the bias term, represents the weight matrix, The eigenvalue of the jth characteristic element representing the sample feature, It represents the eigenvalue of the jth feature element in the sample reconstruction feature, and f() represents the activation function.

[0110] Optionally, the activation function may include at least one of the following: a sigmoid function and a hyperbolic tangent function.

[0111] The weight matrix may include an encoding weight matrix and a decoding weight matrix. For example, Can be refined into .in, Expressed as the encoding weight matrix, is represented as a decoding weight matrix.

[0112] The reconstruction loss function can be expressed as the mean square error between the sample features and the sample reconstruction features, as shown in the following formula (2).

[0113] ;Formula (2)

[0114] in, Represents the reconstruction loss function, j represents the jth feature element, J represents the total number of feature elements, Represents the eigenvalue of the jth feature element in the sample feature, Represents the eigenvalue of the jth feature element in the sample reconstructed feature.

[0115] The regularization loss function can be expressed as follows:

[0116] ;Formula (3)

[0117] in, represents the regularized loss function, k represents the kth iteration round, K represents the total number of iteration rounds, represents the hyperparameter, represents the model parameter matrix of the kth round, represents the regularization parameter of the kth round, Represents the norm of the model parameter matrix of the kth round.

[0118] Model parameter matrix for round k It can be expressed by the following formula (4).

[0119] Optionally, ;Formula (4)

[0120] in, represents the sample scale parameter; represents the adjustable parameter matrix of the kth round affected by the sample scale parameter; The weight matrix representing the kth round may include an encoding weight matrix and a decoding weight matrix.

[0121] It should be noted that the model parameter matrix can be determined based on the weight matrix in the initial model parameters, the sample scale parameter and the adjustable parameter matrix, and the regularization loss function is determined based on the model parameter matrix.

[0122] The objective loss function E can be .

[0123] The iterative optimization of the target loss function can use the gradient descent algorithm, with the convergence of the target loss function as the iteration termination condition.

[0124] Using the reconstruction loss function to adjust the parameters of the sparse autoencoder can ensure that the important features before and after reconstruction remain unchanged, thereby ensuring the invariance of the reconstructed features. In addition, using the regularization loss function to adjust the parameters of the sparse autoencoder can ensure the sparsity of the reconstructed features, thus achieving a sparse constraint effect.

[0125] Furthermore, the sample scale parameters are used as reference data, and the matrix of adaptive adjustable parameters is added and combined with the weight matrix to jointly determine the model parameter matrix. Thus, the adaptability between the feature scale and the model parameters is ensured through the mapping relationship between the feature scale and the model parameters.

[0126] The above article explains how to optimize model parameters. The following article will further explain the specific composition of model parameters in combination with the network structure of sparse autoencoders.

[0127] Figure 5 A flowchart of unsupervised training of a sparse autoencoder according to an embodiment of the present application is shown.

[0128] like Figure 5As shown, the sparse autoencoder may include an encoder M510 and a decoder M520. For example, the initial model parameters may include an encoding weight matrix, a decoding weight matrix, and a bias term. The encoder M510 configured with the encoding weight matrix may be used to encode the sample features 510 to obtain sample encoded features 520. The decoder M520 configured with the decoding weight matrix and the bias term may be used to decode the sample encoded features 520 to obtain sample reconstructed features 530.

[0129] like Figure 5 As shown, based on the sample features 510 and the sample reconstruction features 530 , a reconstruction loss function 540 is determined.

[0130] like Figure 5 As shown, a regularization loss function 560 is determined based on the initial model parameters and a sample scale parameter 550 representing a characteristic scale of the sample features.

[0131] like Figure 5 As shown, based on the reconstruction loss function 540 and the regularization loss function 560, the initial model parameters are adjusted to obtain model parameters that match the sample scale parameters.

[0132] The sparse autoencoder training method provided in the embodiments of this application can achieve sparsity while avoiding the loss of key information, achieving a balance between sparse representation and the preservation of key features. Furthermore, during feature reconstruction, the characteristic scale of the features to be processed will affect the reconstruction effect. Therefore, the feature scale and model parameters are adapted. Based on the mapping relationship between the feature scale and the model parameters, the model parameters are adaptively adjusted to further improve the reconstruction effect.

[0133] The above section explains how to perform feature reconstruction and optimize model parameters. The following section explains how to extract semantic features after reconstruction.

[0134] According to the embodiments of the present application, Figure 2 Operation S240 shown, performing semantic feature extraction on the reconstructed feature to obtain the semantic feature, may include: performing semantic feature extraction on the reconstructed feature using a plurality of cascaded gated recurrent units to obtain the semantic feature.

[0135] The greater the number of cascaded gated recurrent units, the deeper the semantic feature extraction will be, but the less original features will be retained and the longer the processing time will be. To balance semantic features and processing efficiency, the number of cascades can be adjusted to 2.

[0136] GRU is used to further explore the temporal relationship and complex connections between features, enhancing the effect of feature fusion and information interaction, thereby improving the accuracy of object recognition.

[0137] The object recognition method will be explained in detail below through further refinement of the object recognition model.

[0138] Figure 6 A network structure diagram of an object recognition model according to another embodiment of the present application is shown.

[0139] like Figure 6 As shown, the object recognition model may include a pre-processing module M610, multiple GRUs M620 in parallel, a splicing unit M631, a first reconstruction unit SAE M632, a feature extraction unit M633, a second reconstruction unit SAE M634, a fusion unit M635, cascaded GRU M641 and GRU M642, a post-processing module M650, and a classifier M660.

[0140] The preprocessing module M610 can perform target detection on the video signals in the multiple initial time series signals 601 to obtain target detection results. If the target detection results indicate the presence of a target object in the target region within a target time period, signal extraction is performed on each of the multiple initial time series signals based on the target time period to obtain multiple time series signals 602. Multiple GRUs M620 connected in parallel perform time series feature extraction on the multiple time series signals 602 to obtain multiple time series features. The time series features are input into the splicing unit M631 to obtain splicing features 603. The splicing features 603 are input into the first reconstruction unit SAE M632 to obtain initial reconstructed features 604. The initial reconstructed features 604 are input into the feature extraction unit M633 to obtain first reconstructed features 605. The first reconstructed features 605 are input into the second reconstruction unit SAE M634 to obtain second reconstructed features 606. The first reconstructed features 605 and the second reconstructed features 606 are input into the fusion unit M635 to obtain reconstructed features 607. The reconstructed features 607 are input into the cascaded GRU M641 and GRU M642 to obtain semantic features 608. The semantic features 608 are input into the post-processing module M650 to obtain target features 609. The target features 609 are input into the classifier M660 to obtain the recognition result 610.

[0141] According to the embodiments of the present application, Figure 2 Operation S250 shown as post-processing the semantic features to obtain target features may include: performing vector dimension transformation on the semantic features to obtain transformed features, and performing normalization on the transformed features to obtain target features.

[0142] Alternatively, see Figure 6 The post-processing module M650 shown in FIG. 1 may include a transformation unit and a normalization unit. The transformation unit is used to transform the vector dimension of the semantic feature. For example, the three-dimensional The semantic features of the vector are transformed into two dimensions to obtain a two-dimensional The transformed features. Indicates height, Indicates width, Indicates a channel.

[0143] Through the above vector dimension conversion method, features from different channels and positions can be integrated in a simple and direct way, providing a comprehensive feature representation for subsequent global normalization.

[0144] In another example, in the global normalization stage, a specific normalization operation can be used to ensure the stability of the output results under different data input conditions.

[0145] According to an embodiment of the present application, by combining vector dimension transformation and normalization as post-processing means, features can be integrated using vector dimension transformation. In addition, the use of normalization means can make the transformed features have the same dimension, avoiding the problem of poor classification effect due to different dimensions.

[0146] According to the embodiments of the present application, Figure 2 Operation S260 shown, performing object recognition on the target area using the target feature to obtain a recognition result, may include: inputting the target feature into a classifier to obtain a recognition result for characterizing whether the target object in the target area is an abnormal object.

[0147] For example, a classifier can perform a binary classification to determine whether a target object is abnormal. However, this is not limited to this. The classifier can also perform multiple classifications. For example, using this object recognition method in a smart home system, it can classify target features to determine whether changes to the home environment are necessary. For example, if a user experiences discomfort, such as excessively low or high indoor temperatures, the recognition result can be used to control the automatic adjustment of home appliances, such as automatically turning on lights and adjusting the indoor temperature based on the user's movements upon returning home and changes in ambient light.

[0148] By utilizing the multimodal signal fusion and object recognition technology provided in the embodiments of the present application, it is possible to combine multiple time series signals of different modalities to improve the effect of object recognition, thereby expanding the scope of application.

[0149] Based on the above object recognition method, the present application also provides an object recognition device.

[0150] The object recognition device of this embodiment includes a first module, a second module, a third module, a fourth module, a fifth module and a sixth module.

[0151] The first module is configured to acquire multiple time series signals of different modalities about the target area collected by multiple types of sensors. In one embodiment, the first module may be configured to execute the operation S210 described above, which will not be described in detail here.

[0152] The second module is configured to extract time series features from the multiple time series signals to obtain multiple time series features. In one embodiment, the first module may be configured to perform the operation S220 described above, which will not be described in detail here.

[0153] The third module is configured to perform feature reconstruction on the plurality of time series features to obtain reconstructed features. In one embodiment, the first module may be configured to perform the operation S230 described above, which will not be described in detail here.

[0154] The fourth module is configured to extract semantic features from the reconstructed features to obtain semantic features. In one embodiment, the first module may be configured to perform the operation S240 described above, which will not be described in detail here.

[0155] The fifth module is configured to post-process the semantic features to obtain target features. In one embodiment, the first module may be configured to perform the operation S250 described above, which will not be described in detail here.

[0156] The sixth module is configured to perform object recognition on the target area using the target features to obtain a recognition result. In one embodiment, the first module may be configured to perform the operation S260 described above, which will not be described in detail here.

[0157] According to an embodiment of the present application, the third module includes: a first unit, a second unit, a third unit, and a fourth unit.

[0158] The first unit is used to splice multiple time series features to obtain a spliced ​​feature.

[0159] The second unit is used to determine a target model parameter that matches a feature scale of the splicing feature from the multiple model parameters.

[0160] The third unit is used to perform sparsification reconstruction on the spliced ​​features using a sparse autoencoder configured with target model parameters to obtain initial reconstructed features.

[0161] The fourth unit is used to extract features from the initial reconstruction features to obtain reconstruction features.

[0162] According to an embodiment of the present application, the object recognition device further includes: a seventh module, an eighth module, and a ninth module.

[0163] The seventh module is used to use the initial reconstructed features after feature extraction as the first reconstructed features.

[0164] The eighth module is used to perform sparsification reconstruction on the first reconstructed feature by using a sparse autoencoder configured with target model parameters when the feature scales of the first reconstructed feature and the spliced ​​feature are the same, so as to obtain a second reconstructed feature.

[0165] The ninth module is used to fuse the first reconstruction feature and the second reconstruction feature to obtain a reconstruction feature.

[0166] According to an embodiment of the present application, the object recognition device further includes: a tenth module and an eleventh module.

[0167] The tenth module is used to perform feature element filling processing on the first reconstructed feature to obtain a preprocessed reconstructed feature when the feature scales of the first reconstructed feature and the spliced ​​feature are different.

[0168] The eleventh module is used to perform a second sparse reconstruction on the preprocessed reconstruction features using a sparse autoencoder configured with target model parameters to obtain second reconstructed features.

[0169] According to an embodiment of the present application, multiple model parameters are determined as follows: a sparse autoencoder configured with initial model parameters is trained separately using multiple sample features of different feature scales to obtain multiple model parameters; wherein, training the sparse autoencoder configured with initial model parameters using sample features includes the following operations: sparsifying and reconstructing the sample features using the sparse autoencoder configured with the initial model parameters to obtain sample reconstruction features; determining a reconstruction loss function based on the sample features and the sample reconstruction features; determining a regularization loss function based on the initial model parameters and the sample scale parameters of the feature scale characterizing the sample features; and adjusting the initial model parameters based on the reconstruction loss function and the regularization loss function to obtain model parameters that match the sample scale parameters.

[0170] According to an embodiment of the present application, the initial model parameters include a weight matrix and a bias term; the sample features are sparsely reconstructed using a sparse autoencoder configured with the initial model parameters to obtain sample reconstruction features, including: encoding the sample features using a sparse autoencoder to obtain sample encoding features; fusing the sample encoding features with the weight matrix and then splicing them with the bias term to obtain sample sparsification features; and activating the sample sparsification features to obtain sample reconstruction features.

[0171] According to an embodiment of the present application, the first module includes: a fifth unit, a sixth unit, and a seventh unit.

[0172] The fifth unit is used to obtain multiple initial time series signals of different modes about the target area collected by multiple types of sensors.

[0173] The sixth unit is configured to perform target detection on the video signal in the plurality of initial time sequence signals to obtain a target detection result.

[0174] The seventh unit is configured to extract signals from the multiple initial time series signals based on the target time period to obtain multiple time series signals when the target detection result indicates that the target object exists in the target area within the target time period.

[0175] According to an embodiment of the present application, the seventh unit includes: a first subunit and a second subunit.

[0176] The first subunit is configured to intercept the plurality of initial timing signals respectively based on a target time period to obtain signal segments.

[0177] The second subunit is used to extract key signals from signal segments matching the sensor based on the sensor's signal acquisition frequency and the target signal frequency to obtain a timing signal. The target signal frequency is determined based on the signal acquisition frequencies of multiple types of sensors.

[0178] According to an embodiment of the present application, the second module includes: an eighth unit.

[0179] The eighth unit is configured to extract timing features from a timing signal that matches the gated recurrent unit using a plurality of gated recurrent units connected in parallel, thereby obtaining a plurality of timing features.

[0180] According to an embodiment of the present application, the fourth module includes: a ninth unit.

[0181] The ninth unit is used to extract semantic features from the reconstructed features using a plurality of cascaded gated recurrent units to obtain semantic features.

[0182] According to an embodiment of the present application, the fifth module includes: a tenth unit.

[0183] The tenth unit is used to perform vector dimension transformation on the semantic features to obtain transformed features; and perform normalization operation on the transformed features to obtain target features.

[0184] According to an embodiment of the present application, the sixth module includes: an eleventh unit.

[0185] The eleventh unit is used to input the target feature into the classifier to obtain a recognition result for characterizing whether the target object in the target area is an abnormal object.

[0186] According to embodiments of the present application, any multiple modules among the first, second, third, fourth, fifth, and sixth modules may be combined into a single module, or any one of them may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present application, at least one of the first, second, third, fourth, fifth, and sixth modules may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of these. Alternatively, at least one of the first, second, third, fourth, fifth, and sixth modules may be at least partially implemented as a computer program module that, when executed, performs the corresponding functionality.

[0187] Figure 7 A block diagram of an electronic device suitable for implementing an object recognition method according to an embodiment of the present application is shown.

[0188] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0189] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in the one or more memories.

[0190] According to an embodiment of the present application, electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to bus 704. Electronic device 700 may also include one or more of the following components connected to I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 708 including a hard disk; and a communication section 709 including a network interface card such as a LAN card or modem. Communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to I / O interface 705 as needed. Removable media 711, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 710 as needed, so that computer programs read from the removable media can be installed into storage section 708 as needed.

[0191] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0192] According to embodiments of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0193] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the object recognition method provided in the embodiments of the present application.

[0194] The computer program executes the above functions defined in the system / device of the embodiment of the present application when the computer program is executed by the processor 701. According to the embodiment of the present application, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0195] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0196] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0197] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0198] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0199] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

[0200] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A method for object recognition, characterized in that: The method comprises: Acquire multiple time series signals of different modalities about the target area collected by multiple types of sensors; Extracting time series features from the plurality of time series signals to obtain a plurality of time series features; The method comprises the following steps: reconstructing a plurality of the time series features to obtain a reconstructed feature, comprising: splicing the plurality of the time series features to obtain a spliced ​​feature; determining a target model parameter that matches the feature scale of the spliced ​​feature from a plurality of model parameters based on a mapping relationship between feature scales and model parameters; performing sparse reconstruction on the spliced ​​feature using a sparse autoencoder configured with the target model parameters to obtain an initial reconstructed feature; using the initial reconstructed feature after feature extraction as a first reconstructed feature; performing feature element padding processing on the first reconstructed feature when the feature scales of the first reconstructed feature and the spliced ​​feature are different to obtain a preprocessed reconstructed feature; performing sparse reconstruction on the preprocessed reconstructed feature using a sparse autoencoder configured with the target model parameters to obtain a second reconstructed feature; fusing the first reconstructed feature and the second reconstructed feature to obtain a reconstructed feature; and extracting semantic features from the reconstructed feature to obtain a semantic feature. Post-processing the semantic features to obtain target features; and Performing object recognition on the target area using the target features to obtain a recognition result; The model parameters are determined as follows: Using multiple sample features of different feature scales to train a sparse autoencoder configured with initial model parameters to obtain multiple model parameters; The step of training the sparse autoencoder configured with the initial model parameters using the sample features includes the following operations: Using a sparse autoencoder configured with the initial model parameters to perform sparse reconstruction on the sample features to obtain sample reconstructed features; Determining a reconstruction loss function based on the sample features and the sample reconstruction features; Determining a model parameter matrix based on a weight matrix in the initial model parameters, a sample scale parameter representing a characteristic scale of the sample characteristics, and an adjustable parameter matrix; Determining a regularization loss function based on the model parameter matrix; and Based on the reconstruction loss function and the regularization loss function, the initial model parameters are adjusted to obtain model parameters that match the sample scale parameters.

2. The method according to claim 1, characterized in that The method further comprises: When the first reconstructed feature and the spliced ​​feature have the same feature scale, the first reconstructed feature is sparsely reconstructed using a sparse autoencoder configured with the target model parameters to obtain the second reconstructed feature.

3. The method according to claim 1, characterized in that The initial model parameters include a weight matrix and a bias term; The step of performing sparse reconstruction on the sample features by using a sparse autoencoder configured with the initial model parameters to obtain sample reconstructed features includes: Encoding the sample features using the sparse autoencoder to obtain sample encoding features; Fusing the sample coding feature with the weight matrix and then concatenating the feature with the bias term to obtain a sample sparsification feature; and The sample sparsification feature is activated to obtain the sample reconstruction feature.

4. The method according to claim 1, wherein The acquiring of multiple time series signals of different modalities about the target area collected by multiple types of sensors includes: Acquiring a plurality of initial time series signals of different modalities about the target area collected by a plurality of types of sensors; performing target detection on the video signals in the plurality of initial time series signals to obtain a target detection result; and In a case where the target detection result indicates that a target object exists in the target area within a target time period, signal extraction is performed on the multiple initial time series signals based on the target time period to obtain multiple time series signals.

5. The method according to claim 4, characterized in that The extracting the plurality of initial time series signals based on the target time period to obtain the plurality of time series signals comprises: Based on the target time period, respectively intercept the plurality of initial timing signals to obtain a plurality of signal segments; and Based on the signal acquisition frequency of the sensor and the target signal frequency, key signal extraction is performed on the signal segment matching the sensor to obtain the timing signal. The target signal frequency is determined based on the signal acquisition frequencies of multiple types of sensors.

6. The method according to claim 1, wherein The extracting time series features from the plurality of time series signals to obtain a plurality of time series features includes: A plurality of gated recurrent units connected in parallel are used to extract the timing features of a timing signal that matches the gated recurrent unit, thereby obtaining a plurality of the timing features.

7. The method according to claim 1, characterized in that The extracting semantic features from the reconstructed features to obtain semantic features includes: Semantic features are extracted from the reconstructed features using a plurality of cascaded gated recurrent units to obtain the semantic features.

8. The method according to claim 1, characterized in that The post-processing of the semantic features to obtain target features includes: Performing vector dimension transformation on the semantic features to obtain transformed features; and A normalization operation is performed on the transformed features to obtain the target features.

9. The method according to claim 1, characterized in that The performing object recognition on the target area by using the target feature to obtain a recognition result includes: The target feature is input into a classifier to obtain the recognition result for characterizing whether the target object in the target area is an abnormal object.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images