Scene recognition method and device, equipment, storage medium and program product
By combining information from various modalities such as vision, infrared and sound, feature data are extracted and fused to perform scene recognition, the problem of inaccurate identification in complex environments is solved, and a high-precision recognition effect is achieved.
Patent Information
- Application Number
- CN202411632891.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-14
- Publication Date
- 2025-03-04
AI Technical Summary
In the business scenario of rural teacher file management intelligent confidentiality room, existing recognition technology is difficult to provide reliable identification results due to changes in lighting conditions, unsatisfactory camera angles, and partial obstruction of items.
By obtaining the data to be identified, including visual images, infrared images and sounds, these data are input into the feature extraction network, multiple feature data are extracted, and then the fused feature data is determined based on these feature data, and input it into the recognition model for scene recognition.
In complex environments such as light changes, unsatisfactory camera angles, and partial obstruction of items, it is possible to accurately identify people and their items, improving the reliability of scene recognition.
Smart Images
Figure CN119557619A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information processing technology, and in particular to a scene recognition method, device, equipment, storage medium and program product. Background Art
[0002] In the business scenario of the intelligent confidential room for rural teacher archive management, in order to ensure the security of the archives, it is necessary to accurately identify the people entering the confidential room and the items they carry.
[0003] However, due to the problems often encountered in actual application environments, such as changing lighting conditions, non-ideal camera angles, and partial occlusion of objects, existing recognition technologies are difficult to provide reliable recognition results in these complex scenarios. Summary of the invention
[0004] Based on this, it is necessary to provide a scene recognition method, device, equipment, storage medium and program product that can improve the reliability of recognition results in response to the above technical problems.
[0005] In a first aspect, the present application provides a scene recognition method, comprising:
[0006] Acquiring data to be identified, the data to be identified including visual images to be identified, infrared images to be identified, and sounds to be identified;
[0007] Inputting the data to be identified into a feature extraction network to obtain multiple feature data, the feature extraction network includes a visual image feature extraction network, an infrared image feature extraction network and a sound feature extraction network;
[0008] Determine fusion feature data of the data to be identified according to each feature data;
[0009] The fused feature data is input into the recognition model, and scene recognition is performed on the data to be recognized based on the output results of the recognition model.
[0010] In one of the embodiments, determining fused feature data of the data to be identified based on each feature data includes: determining feature fused data of the data to be identified based on each feature data and a corresponding fusion weight.
[0011] In one embodiment, determining feature fusion data of the data to be identified according to each feature data and the corresponding fusion weight includes:
[0012] The data to be identified is input into the context weight network; the fusion weight corresponding to each feature data is determined according to the context weight network; feature fusion is performed according to the product of each feature data and the corresponding fusion weight to determine the feature fusion data of the data to be identified.
[0013] In one of the embodiments, feature fusion data of the data to be identified is determined based on each feature data and the corresponding fusion weight, and also includes: obtaining the fusion weight corresponding to each feature data; performing feature fusion according to the product of each feature data and the corresponding fusion weight to determine initial feature fusion data; and determining the feature fusion data of the data to be identified based on the initial feature fusion data and the hybrid attention mechanism.
[0014] In one of the embodiments, the data to be identified also includes scene data, and the feature extraction network also includes a self-supervised feature extraction network trained based on a self-supervised learning mechanism, and the self-supervised feature extraction network is used to determine feature data of the scene data.
[0015] In one of the embodiments, scene recognition is performed on the data to be recognized based on the output results of the recognition model, including: identifying the identities of people in the scene, identifying the operating actions of people in the scene, identifying illegal objects in the scene, and / or identifying the location or data of objects in the scene; if there is an abnormality in the scene recognition, an alarm message is sent.
[0016] In a second aspect, the present application further provides a scene recognition device, comprising:
[0017] An acquisition module is used to acquire data to be identified, wherein the data to be identified includes visual images to be identified, infrared images to be identified, and sounds to be identified;
[0018] A feature extraction module is used to input the data to be identified into a visual image feature extraction network, an infrared image feature extraction network and a sound feature extraction network respectively to obtain a plurality of feature data;
[0019] A fusion module, used for determining fusion feature data of the data to be identified according to each feature data;
[0020] The recognition module is used to input the fused feature data into the recognition model and perform scene recognition on the data to be recognized according to the output results of the recognition model.
[0021] In one of the embodiments, the fusion module is specifically used to determine feature fusion data of the data to be identified according to each feature data and the corresponding fusion weight.
[0022] In one of the embodiments, the fusion module is specifically used to input the data to be identified into the context weight network; determine the fusion weight corresponding to each feature data according to the context weight network; perform feature fusion according to the product of each feature data and the corresponding fusion weight, and determine the feature fusion data of the data to be identified.
[0023] In one of the embodiments, the fusion module is specifically used to obtain the fusion weight corresponding to each feature data; perform feature fusion according to the product of each feature data and the corresponding fusion weight to determine the initial feature fusion data; determine the feature fusion data of the data to be identified according to the initial feature fusion data and the hybrid attention mechanism.
[0024] In one of the embodiments, the data to be identified also includes scene data, and the feature extraction network also includes a self-supervised feature extraction network trained based on a self-supervised learning mechanism, and the self-supervised feature extraction network is used to determine feature data of the scene data.
[0025] In one of the embodiments, the identification module is used to identify the identity of people in the scene, identify the operating actions of people in the scene, identify illegal objects in the scene, and / or identify the location or data of objects in the scene based on the output results; if there is an abnormality in the scene identification, an alarm message is sent.
[0026] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, any of the methods described in the first aspect is implemented.
[0027] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the methods described in the first aspect above.
[0028] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the first aspect above.
[0029] The above-mentioned scene recognition method, device, equipment, storage medium and program product obtain the data to be recognized, which includes the visual image to be recognized, the infrared image to be recognized and the sound to be recognized, and then input the data to be recognized into the feature extraction network to obtain multiple feature data, the feature extraction network includes the visual image feature extraction network, the infrared image feature extraction network and the sound feature extraction network, and then determine the fusion feature data of the data to be recognized according to each feature data, and finally input the fusion feature data into the recognition model, and perform scene recognition on the data to be recognized according to the output result of the recognition model. In this way, by combining information of multiple modes such as vision, infrared and sound, it is possible to accurately identify people and the items they carry in complex environments such as changes in lighting, undesirable camera angles and partial occlusion of objects, that is, it is still possible to maintain high-precision recognition capabilities in complex environments, thereby improving the reliability of scene recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0031] FIG1 is a diagram of an application environment of a scene recognition method according to an embodiment;
[0032] FIG2 is a schematic diagram of a flow chart of a scene recognition method in one embodiment;
[0033] FIG3 is a schematic diagram of a flow chart of a step of determining feature fusion data of data to be identified in one embodiment;
[0034] FIG4 is a schematic diagram of a flow chart of a step of determining feature fusion data of data to be identified in another embodiment;
[0035] FIG5 is a schematic diagram of a flow chart of a scene recognition step in one embodiment;
[0036] FIG6 is a schematic flow chart of a scene recognition method in another embodiment;
[0037] FIG7 is a structural block diagram of a scene recognition device in one embodiment;
[0038] FIG. 8 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0040] The scene recognition method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the data to be identified, and the data to be identified includes a visual image to be identified, an infrared image to be identified, and a sound to be identified. Then, the data to be identified is input into the feature extraction network to obtain multiple feature data. The feature extraction network includes a visual image feature extraction network, an infrared image feature extraction network, and a sound feature extraction network. Then, the fused feature data of the data to be identified is determined according to each feature data. Finally, the fused feature data is input into the recognition model, and the scene recognition is performed on the data to be identified according to the output result of the recognition model. The scene recognition result can be sent to the terminal 102 later, wherein the terminal 102 can be, but not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head mounted device, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0041] In an exemplary embodiment, as shown in FIG2 , a scene recognition method is provided, which is described by taking the method applied to the server in FIG1 as an example, and includes the following steps 201 to 204. Among them:
[0042] S201, obtaining data to be identified.
[0043] The data to be identified include visual images to be identified, infrared images to be identified and sounds to be identified.
[0044] Optionally, surveillance video of the confidential room can be obtained through pre-set monitoring equipment, and the surveillance video may include images and sounds. At the same time, the monitoring equipment can be an infrared thermal imaging camera with audio acquisition function, a monitoring camera with infrared fill light, etc., through which the visual image, infrared image and sound of the confidential room can be collected simultaneously.
[0045] Optionally, a visual image can be an image formed through the perception and recording of visible light, which usually has a high resolution and rich color information, and can truly reflect the appearance, shape, color and other characteristics of the object; an infrared image is an image formed by an infrared scanner receiving and recording the thermal radiation energy emitted by the target object. It is a grayscale image and has no color or shadow, but can reflect the temperature distribution of the object, is not affected by visible light, and can detect targets that are obscured or hidden behind the object.
[0046] Optionally, the visual image to be identified, the infrared image to be identified, and the sound to be identified may be data collected by a monitoring device, and the monitoring device may send the collected data to the server in a wired or wireless manner. It is understandable that the server may continuously and in real time acquire the data to be identified, or acquire the data to be identified according to a preset period, or acquire the data to be identified for a preset period of time when a person is detected to enter the confidential room, or acquire the historically collected data from a database for identification, which is not limited in the embodiments of the present application.
[0047] S202, inputting the data to be identified into a feature extraction network to obtain a plurality of feature data.
[0048] Among them, the feature extraction network includes a visual image feature extraction network, an infrared image feature extraction network and a sound feature extraction network.
[0049] Optionally, the feature extraction network may include multiple branches, and according to the difference of the data to be identified, features of the data to be identified are extracted through different branches.
[0050] Optionally, the visual feature extraction network can be a convolutional neural network, such as LeNet, AlexNet, VGGNet, ResNet, InceptionNet or DenseNet, or a network based on an autoencoder, or a network based on a variational autoencoder, or features are extracted through YOLO or Mask R-CNN, or a network implemented based on a histogram of oriented gradients, scale-invariant feature transform or local binary pattern.
[0051] Optionally, the infrared image feature extraction network may include a network that is fine-tuned from a pre-trained model, that is, utilizing a deep learning model pre-trained on a large-scale image dataset, and adjusting the parameters of the deep learning model according to the characteristics of the infrared image, so that the model adapts to the characteristic distribution of the infrared image; it may also be a network implemented based on temperature histogram, temperature gradient features, grayscale co-occurrence matrix, edge detection or contour feature description.
[0052] Optionally, the sound to be recognized can be directly taken as input and then input into the feature extraction network, or the sound to be recognized can be converted into text data and then the text data can be input into the feature extraction network; the sound feature extraction network can be a network based on Mel-frequency cepstral coefficients, or it can be a convolutional neural network, a deep neural network, a recurrent neural network, a long short-term memory network, a gated recurrent unit or a generative adversarial network, etc.
[0053] S203: Determine fused feature data of the data to be identified based on each feature data.
[0054] Optionally, the fused feature data can be the integration of feature data of different modalities, so as to utilize the advantages of each modality and improve the recognition accuracy of subsequent recognition models. For example, the feature data of the infrared image to be recognized can supplement the visual information in insufficient light, and the feature data of the sound to be recognized can be used to identify sound events that are not easy to capture visually.
[0055] In a possible implementation manner, the fused feature data of the data to be identified may be determined by each feature data and a weight corresponding to each feature data.
[0056] In another possible implementation method, each feature data may be combined into a new feature matrix, and then dimension reduction is performed through methods such as principal component analysis or linear discriminant analysis to obtain fused feature data.
[0057] In another possible implementation, multi-layer perceptrons (MLP) or deep convolutional networks can be used to fuse data from different modalities. These networks can learn common representations across modalities while maintaining the independent characteristics between modalities.
[0058] S204, inputting the fused feature data into a recognition model, and performing scene recognition on the data to be recognized according to the output result of the recognition model.
[0059] Optionally, the fused feature data is used as input, and the fused feature data is recognized by a recognition model to determine a recognition result.
[0060] Optionally, the recognition model can be implemented based on a support vector machine, a decision tree, etc., or it can be a multimodal fusion Transformer model, or it can be a Bayesian network fusion model or a dynamic Bayesian network fusion model.
[0061] Optionally, the object status, personnel information and personnel activity information in the scene to be identified can be determined according to the output results of the recognition model to achieve scene recognition.
[0062] The above scene recognition method obtains the data to be recognized, which includes the visual image to be recognized, the infrared image to be recognized and the sound to be recognized. Then, the data to be recognized is input into the feature extraction network to obtain multiple feature data. The feature extraction network includes the visual image feature extraction network, the infrared image feature extraction network and the sound feature extraction network. Then, the fused feature data of the data to be recognized is determined according to each feature data. Finally, the fused feature data is input into the recognition model, and the scene recognition is performed on the data to be recognized according to the output result of the recognition model. In this way, by combining information of multiple modalities such as vision, infrared and sound, it is possible to accurately identify people and the items they carry in complex environments such as changes in lighting, undesirable camera angles and partial occlusion of objects, that is, it is possible to maintain high-precision recognition capabilities in complex environments, thereby improving the reliability of scene recognition.
[0063] In an exemplary embodiment, optionally, determining fused feature data of the data to be identified according to each feature data includes:
[0064] The feature fusion data of the data to be identified is determined according to each feature data and the corresponding fusion weight.
[0065] Optionally, the feature fusion data of the data to be identified can be determined according to the following formula:
[0066]
[0067] Among them, F combined is the feature fusion data, V f IR f and S f are the feature data of visual image, infrared image and sound respectively, and W v , W ir and W s is the weight corresponding to each feature data, b is biased, Represents feature fusion operations, which can be weighted sum, concatenation, etc. f =f v (V), IR f =f ir (IR), S f =f s (S), V, IR, S represent visual image, infrared image and sound input data respectively; f v 、f ir and f s They are the feature extraction functions corresponding to the input data.
[0068] In a possible implementation, as shown in FIG3 , determining feature fusion data of the data to be identified according to each feature data and the corresponding fusion weight includes the following steps 301 to 303 . Among them:
[0069] S301, inputting the data to be identified into the context weight network.
[0070] Optionally, during the training process of the context weight network, the fusion weight parameters are optimized together with other parameters of the network (such as multi-head attention parameters in Transformer, convolution kernel parameters in CNN, etc.). The back propagation algorithm can be used to derive the fusion weight parameters according to the loss function, and adjust the fusion weights so that the network can better predict based on the context.
[0071] S302, determining the fusion weight corresponding to each feature data according to the context weight network.
[0072] Optionally, the context weight network can be an adaptive mechanism, a network architecture that can dynamically assign weights based on the context information of the input data, which can recognize that the context around the data point (such as previous and next data elements or adjacent data elements, etc.) has an important impact on the understanding and processing of the data point, and this impact should have different weights in different situations (normal light environment, noisy environment or night environment).
[0073] Optionally, the fusion weight corresponding to each feature data can be determined according to the context weight network, for example, reducing the weight of sound data in a noisy environment, or increasing the weight of infrared images in a night environment.
[0074] Optionally, the fusion weight of each feature data can be determined by:
[0075]
[0076] in, are the fusion weights corresponding to each feature data, W c and b c are the parameters of the context weight network.
[0077] S303, performing feature fusion according to the product of each feature data and the corresponding fusion weight, and determining feature fusion data of the data to be identified.
[0078] Optionally, after determining the fusion weight corresponding to each feature data, feature fusion data can be obtained by performing feature fusion on the product of each feature data and the corresponding fusion weight, which can be achieved by the following formula:
[0079]
[0080] Above It may be fused feature data calculated by the fusion weight determined by the adaptive mechanism and each feature data.
[0081] The above context weight network dynamically adjusts the fusion strategy according to different situations, making the scene recognition method more flexible and stable, and able to adapt to changing environmental conditions.
[0082] In another possible implementation, as shown in FIG4 , determining the feature fusion data of the data to be identified according to each feature data and the corresponding fusion weight also includes the following steps 401 to 403. Among them:
[0083] S401, obtaining the fusion weight corresponding to each feature data.
[0084] In one possible implementation, the fusion weight may be determined based on historical data and preset, and the feature fusion data obtained based on the preset fusion weight may also be adapted to scene recognition in different situations in a subsequent recognition model.
[0085] In another possible implementation method, the fusion weight of each feature data can be determined by principal component analysis. The feature data can be concatenated to obtain a joint feature matrix. PCA analysis is performed on the joint feature matrix to obtain eigenvalues and eigenvectors, where the eigenvalue represents the variance of each principal component. After normalization, the eigenvalue can be used as the weight of the corresponding modal data in the fusion.
[0086] In another possible implementation method, the fusion weight can be determined by the mutual information between different modal data, that is, the mutual information is calculated according to the joint probability distribution and marginal probability distribution of the feature data, and the fusion weight of each modality is determined by the relative size of the mutual information, for example, the fusion weight is allocated by the ratio of the mutual information to the total mutual information.
[0087] S402, performing feature fusion according to the product of each feature data and the corresponding fusion weight to determine initial feature fusion data.
[0088] Optionally, the method for determining the initial feature fusion data is the same as the above F combined The determination method is consistent with that of the embodiment of the present application, and will not be repeated here.
[0089] S403, determining feature fusion data of the data to be identified according to the initial feature fusion data and the hybrid attention mechanism.
[0090] Optionally, the hybrid attention mechanism can include channel attention and spatial attention. Spatial attention can dynamically adjust weights according to the spatial information of the input feature map, and channel attention can dynamically adjust weights according to the importance of each channel. It can perform more fine-grained rotations between different spatial regions and feature channels of the feature map, thereby further enhancing the expressiveness of the features. Especially in multimodal data, the effective information contributed by some modalities in different scenarios is different, and this mechanism can maximize the use of this information.
[0091] Optionally, feature fusion data can be determined in parallel, that is, different types of attention mechanisms are placed in parallel, and then the attention weights calculated by each of them are fused; feature fusion data can also be determined in serial, that is, different types of attention mechanisms are applied in sequence in a certain order. For example, channel attention is first used to perform a preliminary screening of the data to determine the importance of different channels, and then spatial attention is applied on the basis of channel weighting to further focus on key areas in the space.
[0092] Exemplarily, the feature fusion data of the data to be identified can be determined in the following manner:
[0093]
[0094] in, is the initial feature fusion data, It is the feature fusion data determined based on the hybrid attention mechanism, SpatialAttention() is the spatial attention, and ChannelAttention() is the channel attention mechanism.
[0095] By fusing multimodal data and flexibly adjusting the fusion weights of different modal data, important modal features can be obtained. In the subsequent recognition process, the recognition accuracy can be improved, and the adaptability to environmental changes is enhanced, thereby ensuring the reliability of scene recognition.
[0096] In an exemplary embodiment, optionally, the data to be identified also includes scene data, and the feature extraction network also includes a self-supervised feature extraction network trained based on a self-supervised learning mechanism, and the self-supervised feature extraction network is used to determine feature data of the scene data.
[0097] Optionally, in order to enhance the generalization ability of the scene recognition method in different environments, a self-supervised learning mechanism can be introduced to perform pre-training on unlabeled data.
[0098] Optionally, the self-supervised learning mechanism allows the feature extraction network to learn useful features from unlabeled data by creating proxy tasks, such as predicting the similarity between features or reconstructing input data. The self-supervised loss function of the self-supervised learning mechanism is as follows:
[0099]
[0100] Among them, M is the number of types of input data, that is, the number of modes, is the feature extraction function associated with each modality, is visual image data, infrared image data or sound data, is the scene data, It is the feature data generated by the self-supervised task, and MSE represents the mean square error.
[0101] Optionally, the scene data may be other data besides visual images, infrared images and sounds, such as light data.
[0102] Optionally, the self-supervised learning mechanism can also be used in subsequent feature fusion and recognition models by dynamically adjusting the convolution kernel, stride and adaptive network structure, as well as the mechanism of real-time adjustment of the recognition strategy according to environmental changes during the recognition process.
[0103] In an exemplary embodiment, optionally, as shown in FIG5 , performing scene recognition on the data to be recognized according to the output result of the recognition model includes the following steps 501 to 502 . Among them:
[0104] S501, identifying the identities of people in the scene, identifying the operating actions of people in the scene, identifying illegal objects in the scene, and / or identifying the locations or data of objects in the scene based on the output results.
[0105] Optionally, identification of personnel may include face recognition. When a person enters a confidential room, a scene recognition method may accurately identify the person and detect items the person carries, especially illegal communication tools or other prohibited items.
[0106] Optionally, the identification of operating actions of personnel in the scene can be carried out during the process of inventorying and organizing files in the confidential room. The scene recognition method needs to detect and identify whether the operation complies with regulations and whether the equipment is used correctly.
[0107] Optionally, the identification of illegal items in the scene can be carried out by monitoring whether people in the confidential room use communication tools such as mobile phones and laptops, especially in areas where the use of communication equipment is strictly prohibited.
[0108] Optionally, identifying the location or data of items in the scene may be to take inventory of the items in the smart confidential room, and compare the number of items and the location of each item with a preset number and preset location.
[0109] Optionally, the recognition model may be trained according to the above-mentioned scene recognition types, so that the scene recognition model can recognize at least one of the above-mentioned scenes.
[0110] S502: If there is an abnormality in scene recognition, an alarm message is sent.
[0111] Optionally, when there is an abnormality in scene recognition, it may be that the person entering the confidential room is not an authorized person, or the operating action is illegal, or illegal items are carried or used, or the position or data of objects in the scene are abnormal, etc.
[0112] Optionally, when an abnormality occurs, an alarm message may be generated according to a preset alarm format and sent to a management staff member for timely processing.
[0113] Optionally, the alarm information can be sent to the management personnel through communication alarm, sound and light alarm or vibration alarm.
[0114] As an optional implementation, as shown in FIG6 , the scene recognition method provided in the embodiment of the present application may include the following specific steps:
[0115] S601, obtaining data to be identified.
[0116] The data to be identified include visual images to be identified, infrared images to be identified and sounds to be identified.
[0117] S602, inputting the data to be identified into a feature extraction network to obtain a plurality of feature data.
[0118] Among them, the feature extraction network includes a visual image feature extraction network, an infrared image feature extraction network and a sound feature extraction network.
[0119] S603: Input the data to be identified into the context weight network.
[0120] S604: Determine the fusion weight corresponding to each feature data according to the context weight network.
[0121] S605, performing feature fusion according to the product of each feature data and the corresponding fusion weight, and determining feature fusion data of the data to be identified.
[0122] S606, obtaining the fusion weight corresponding to each feature data.
[0123] S607, performing feature fusion according to the product of each feature data and the corresponding fusion weight to determine initial feature fusion data.
[0124] S608, determining feature fusion data of the data to be identified based on the initial feature fusion data and the hybrid attention mechanism.
[0125] S609, inputting the fused feature data into the recognition model.
[0126] S610, identifying the identities of people in the scene, identifying the operating actions of people in the scene, identifying illegal objects in the scene, and / or identifying the locations or data of objects in the scene based on the output results of the recognition model.
[0127] S611: If there is an abnormality in scene recognition, an alarm message is sent.
[0128] In an exemplary embodiment, optionally, the training process of the model used in the above scene recognition method includes the following steps:
[0129] First, the dataset collection process: This dataset focuses on indoor scenes and includes two main modalities: visual images (RGB images) and infrared images (IR images), as well as an auxiliary modality, sound (audio clips). The dataset covers a variety of complex indoor environments and is designed to test the recognition performance of the model under different lighting and noise conditions. For example, the dataset contains a total of 4,000 samples, divided into three scene types (bright indoor scenes, dim indoor scenes, and noisy indoor scenes), and each scene type has 1,333 samples (one of the scene types contains an additional sample).
[0130] Well-lit Indoor Scene: Visual image (RGB image), image resolution 1280x720, well-lit indoor scene, background including furniture, electronic equipment and people, all details are clearly visible; infrared image (IR image), image resolution 1280x720, due to the well-lit, the information provided by the IR image is similar to that of the RGB image, but it is mainly used to capture the thermal signature of objects and the outline of people; sound (audio clip), audio length 5 seconds, low background noise, mainly including people talking and slight environmental sounds (such as fan sound).
[0131] Dimly-lit Indoor Scene: Visual image (RGB image), image resolution 1280x720, indoor scene with insufficient light, blurred details of faces and objects, and obvious shadows in some areas; Infrared image (IR image): Image resolution 1280x720. In a dim environment, IR images become the main source of information, providing clear contours and thermal signals to make up for the missing details in RGB images; Sound (audio clip): Audio length 5 seconds, low-noise environment, but occasionally there will be slight background sounds, such as equipment operation sounds and environmental sound effects.
[0132] Noisy Indoor Scene: Visual image (RGB image), image resolution 1280x720, indoor scene with moderate lighting conditions, but complex background, including multiple people, moving objects and electronic equipment; infrared image (IR image), image resolution 1280x720, in this environment, IR image can still provide key contour information to help identify partially obscured or moving objects; sound (audio clip), audio length 5 seconds, high background noise, including multiple people talking, electronic equipment running, TV or radio sound, voice signal may be drowned by noise.
[0133] Second, the dataset annotation process: Each sample is annotated in detail: in terms of face annotation, the bounding box of the face and its attributes (such as posture, expression, etc.) are annotated in RGB and IR images; in terms of object annotation: the location and category of key objects (such as mobile phones, laptops, bags), as well as the status of objects (such as whether they are carried or visible) are annotated; in terms of sound annotation: the main sound events in the audio clips are annotated, such as conversations and the source of background noise (such as electrical appliances, fans).
[0134] Third, the data set division process: training set: 70% (2800 samples), used for model training. Validation set: 15% (600 samples), used for model parameter adjustment and verification. Test set: 15% (600 samples), used to evaluate the final performance of the model.
[0135] Fourth, model training process:
[0136] Scene 1: Normal lighting environment.
[0137] Sample data: 1. Five teachers entered the confidential room, carrying laptops, mobile phones, folders, bags and other items; 2. The camera was set at a height of 2 meters, with a vertical downward angle; 3. The lighting conditions were normal indoor lighting (300-500 lux).
[0138] Expected Results: The scene recognition method should accurately identify the identity of each teacher and all items they carry, and record the time of entry and the type of items.
[0139] Scene 2: Low light environment.
[0140] Sample data: 1. Three teachers entered the confidential room, carrying a mobile phone, a folder and a small bag respectively; 2. The camera was set at a height of 2.5 meters, with a slightly tilted angle and partially blocked by objects; 3. The lighting condition was a low-light environment (<100 lux); 4. The infrared camera was turned on to capture temperature information.
[0141] Expected Results: The scene recognition method should be combined with infrared data to accurately identify each teacher and the items they carry, detect mobile phones in bags even when partially blocked, and issue a warning.
[0142] Scenario 3: Complex background environment.
[0143] Sample data: 1. Two teachers entered the confidential room at the same time, one of them carried a file bag and the other carried a mobile phone and headphones; 2. The camera angle was high (45° looking down), and some backgrounds had office equipment and sundries, which might cause interference; 3. The background noise was high, including conversation sounds and equipment operation sounds; 4. Sound data was collected at the same time to assist in judgment.
[0144] Expected results: The scene recognition method should accurately separate the features of the target teacher and his / her belongings under complex background interference, detect the ringing of mobile phones in background noise, and issue a warning.
[0145] Scenario 4: Real-time online learning scenario.
[0146] Sample data: 1. Set the simulation to run for 12 hours, during which the lighting conditions, number of people and types of items change randomly; 2. The camera angle is adjusted randomly, and occlusion or perspective changes may occur; 3. The system needs to gradually optimize the recognition model through an online learning mechanism to adapt to environmental changes.
[0147] Expected results: The scene recognition method should maintain high-precision recognition (>90%) during continuous operation, and gradually improve the recognition performance through online learning to reduce misrecognition caused by environmental changes.
[0148] Fifth, comparison and verification process:
[0149] Multimodal fusion effect test: Test the performance of a single modality (such as RGB only, IR only) and the effect of multimodal fusion. Especially in dim and noisy environments, observe how the model uses multimodal data (RGB + IR + sound) to improve recognition accuracy. Compare the performance of the model after integrating the attention mechanism in different scenarios to verify its advantages in adaptability.
[0150] Attention mechanism effect test: Evaluate the performance of the hybrid attention mechanism in different scenarios, especially in dimly lit rooms and noisy environments, and observe how the attention mechanism adjusts the weights of each modality to optimize the recognition results. Compare the baseline model without the attention mechanism with the improved model after using the attention mechanism, and analyze the contribution of the attention mechanism in improving the model performance.
[0151] Self-supervised learning effect test: introduce self-supervised learning mechanism, use unlabeled data for pre-training, and observe its improvement in the generalization ability of the model in dim and noisy scenes. Test the adaptability of self-supervised learning to new scenes (such as unseen indoor lighting changes) and verify its effect in practical applications.
[0152] The performance of the scene recognition method in the embodiment of the present application is as follows:
[0153] Bright Indoor Scene: Simulates a well-lit environment during the day. The recognition accuracy under normal lighting conditions reaches 95%, especially for unobstructed objects and faces, the accuracy is as high as 98%.
[0154] Dim indoor scene: simulates nighttime or low-light environments, and performs well in low-light conditions: with the addition of infrared modality, the recognition accuracy is increased to 88%, of which the recognition accuracy of carried objects reaches 90%.
[0155] Noisy Indoor Scene: Simulates an environment with noisy human voices or loud background noise, with an accuracy rate of 85% in responding to sound modalities and recognizing non-visual events. In particular, the recognition rate for detecting mobile phone ringtones and conversation sounds is 92%.
[0156] Light change scenario: simulates the process of light gradually changing from bright to dark. Due to the introduction of the self-supervised learning mechanism, the recognition accuracy in light changes is increased by 10%, from 80% to 90%.
[0157] The improved model has significantly improved accuracy in all scenarios, especially in low-light and noisy environments, and the performance is more outstanding. The introduction of the attention mechanism enables the model to effectively allocate attention in different modes, ensuring that key features are prioritized. The introduction of the self-supervised learning strategy enables the model to have stronger adaptability when dealing with new scenes or changes in data distribution. The use of unlabeled data expands the learning scope of the model and significantly enhances its generalization ability in complex scenes. Through the hybrid attention mechanism and adaptive feature fusion, the improved model performs more stably under different conditions, not only performing well in bright indoor scenes, but also maintaining efficient recognition capabilities in dim and noisy environments. Through these experimental data and comparative analysis, the superiority of the scene recognition method in the embodiment of the present application in multimodal recognition tasks in complex indoor environments is clearly demonstrated.
[0158] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0159] Based on the same inventive concept, the embodiment of the present application also provides a scene recognition device for implementing the scene recognition method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more scene recognition device embodiments provided below can refer to the limitations of the scene recognition method above, and will not be repeated here.
[0160] In an exemplary embodiment, as shown in FIG. 7 , a scene recognition device 700 is provided, comprising: an acquisition module 701 , a feature extraction module 702 , a fusion module 703 and a recognition module 704 , wherein:
[0161] An acquisition module 701 is used to acquire data to be identified, wherein the data to be identified includes a visual image to be identified, an infrared image to be identified, and a sound to be identified;
[0162] A feature extraction module 702 is used to input the data to be identified into a visual image feature extraction network, an infrared image feature extraction network and a sound feature extraction network respectively to obtain a plurality of feature data;
[0163] A fusion module 703, used to determine fusion feature data of the data to be identified according to each feature data;
[0164] The recognition module 704 is used to input the fused feature data into a recognition model, and perform scene recognition on the data to be recognized according to the output result of the recognition model.
[0165] In one embodiment, the fusion module 703 is specifically configured to determine feature fusion data of the data to be identified according to each feature data and the corresponding fusion weight.
[0166] In one embodiment, the fusion module 703 is specifically used to input the data to be identified into the context weight network; determine the fusion weight corresponding to each feature data according to the context weight network; perform feature fusion according to the product of each feature data and the corresponding fusion weight, and determine the feature fusion data of the data to be identified.
[0167] In one embodiment, the fusion module 703 is specifically used to obtain the fusion weight corresponding to each feature data; perform feature fusion according to the product of each feature data and the corresponding fusion weight to determine the initial feature fusion data; determine the feature fusion data of the data to be identified according to the initial feature fusion data and the hybrid attention mechanism.
[0168] In one of the embodiments, the data to be identified also includes scene data, and the feature extraction network also includes a self-supervised feature extraction network trained based on a self-supervised learning mechanism, and the self-supervised feature extraction network is used to determine feature data of the scene data.
[0169] In one embodiment, the identification module 704 is used to identify the identity of people in the scene, identify the operating actions of people in the scene, identify illegal objects in the scene, and / or identify the location or data of objects in the scene based on the output results; if there is an abnormality in the scene identification, an alarm message is sent.
[0170] Each module in the above scene recognition device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0171] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a scene recognition method is implemented.
[0172] Those skilled in the art will understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0173] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps described in any of the above method embodiments when executing the computer program.
[0174] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps described in any of the above method embodiments are implemented.
[0175] In one embodiment, a computer program product is provided, including a computer program, which implements the steps described in any of the above method embodiments when executed by a processor.
[0176] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0177] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0178] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A scene recognition method, characterized in that: The method comprises: Acquiring data to be identified, wherein the data to be identified includes a visual image to be identified, an infrared image to be identified, and a sound to be identified; Inputting the data to be identified into a feature extraction network to obtain a plurality of feature data, wherein the feature extraction network includes a visual image feature extraction network, an infrared image feature extraction network and a sound feature extraction network; Determine fusion feature data of the data to be identified according to each of the feature data; The fused feature data is input into a recognition model, and scene recognition is performed on the data to be recognized according to an output result of the recognition model.
2. The method according to claim 1, characterized in that The step of determining the fused feature data of the data to be identified according to each of the feature data comprises: The feature fusion data of the data to be identified is determined according to each of the feature data and the corresponding fusion weight.
3. The method according to claim 2, characterized in that The step of determining the feature fusion data of the data to be identified according to each of the feature data and the corresponding fusion weights includes: Inputting the data to be identified into a context weight network; Determine the fusion weight corresponding to each of the feature data according to the context weight network; Feature fusion is performed according to the product of each of the feature data and the corresponding fusion weight to determine the feature fusion data of the data to be identified.
4. The method according to claim 2, characterized in that: The step of determining the feature fusion data of the data to be identified according to each of the feature data and the corresponding fusion weights also includes: Obtaining a fusion weight corresponding to each of the feature data; Perform feature fusion according to the product of each feature data and the corresponding fusion weight to determine initial feature fusion data; Determine the feature fusion data of the data to be identified according to the initial feature fusion data and the hybrid attention mechanism.
5. The method according to claim 1, characterized in that The data to be identified also includes scene data, and the feature extraction network also includes a self-supervised feature extraction network trained based on a self-supervised learning mechanism, and the self-supervised feature extraction network is used to determine the feature data of the scene data.
6. The method according to any one of claims 1 to 4, characterized in that: The performing scene recognition on the data to be recognized according to the output result of the recognition model includes: According to the output result, the identity of the person in the scene is identified, the operation action of the person in the scene is identified, the illegal object in the scene is identified, and / or the location or data of the object in the scene is identified; If there is an abnormality in the scene recognition, an alarm message is sent.
7. A scene recognition device, characterized in that: The device comprises: An acquisition module, used for acquiring data to be identified, wherein the data to be identified includes a visual image to be identified, an infrared image to be identified, and a sound to be identified; A feature extraction module, used for inputting the data to be identified into a visual image feature extraction network, an infrared image feature extraction network and a sound feature extraction network respectively, to obtain a plurality of feature data; A fusion module, used for determining fusion feature data of the data to be identified according to each feature data; The recognition module is used to input the fused feature data into a recognition model and perform scene recognition on the data to be recognized according to the output result of the recognition model.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.