Elevator scene recognition method, device, electronic equipment and storage medium

By using a deep network of action recognition in elevator scene recognition, combined with spatial and temporal feature extraction, the problems of elevator scene recognition error and full-scene monitoring are solved, and the precise recognition of various normal and abnormal scenes in elevators is achieved.

CN114519831BActive Publication Date: 2024-11-19SHENZHEN XUMI YUNTU SPACE TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210144001.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-17
Publication Date
2024-11-19
Estimated Expiration
2042-02-17

AI Technical Summary

Technical Problem

The existing technology has large identification errors in elevator scene recognition, which cannot realize full-scene recognition and monitoring, especially in elevator scenes and passenger actions with high information complexity, diversity within classes and high inter-class similarity. Identify abnormal situations.

Method used

Using the elevator scene recognition method based on the action recognition deep network, the video stream collected by the elevator camera is obtained, and the video clips are extracted by using the action recognition deep network containing three feature extraction channels, combining spatial dimensions and The time dimension features are globally averaged pooled and feature splicing to predict elevator scenes.

Benefits of technology

The recognition error of action scenes is reduced, and the accurate recognition of complex abnormal scenes is achieved, achieving the effect of full-scene recognition and monitoring of elevators.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114519831B_ABST
    Figure CN114519831B_ABST
Patent Text Reader

Abstract

The present disclosure provides an elevator scene recognition method, device, electronic device and storage medium. The method includes: obtaining a video clip obtained by segmenting a video stream according to a preset time length, inputting the video clip into an elevator scene recognition model, and extracting features of the video clip using a motion recognition deep network; extracting spatial dimension features using a first feature extraction channel to obtain a first feature map, extracting temporal dimension features using a second feature extraction channel to obtain a second feature map, and fusing spatial dimension features and temporal dimension features using a third feature extraction channel to obtain a third feature map; performing global average pooling and feature splicing on the first feature map, the second feature map and the third feature map, predicting the feature map obtained after feature splicing, and determining the elevator scene corresponding to the video clip according to the prediction result. The present disclosure can accurately identify elevator scenes and realize full-scene recognition and monitoring of elevators.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an elevator scene recognition method, device, electronic equipment and storage medium. Background Art

[0002] With the development of computer technology and multimedia technology, the application of scene monitoring using video is becoming more and more widespread. For example, by monitoring the scene inside an elevator, abnormal situations can be discovered and responded to in a timely manner.

[0003] In the prior art, the use of traditional video surveillance systems requires real-time monitoring of surveillance videos, which is prone to misjudgment and missed judgments. Although some artificial intelligence algorithms based on computer vision can identify behaviors in videos, in the process of elevator scene recognition, the elevator scenes and passenger actions that need to be identified have high information complexity, intra-class diversity and inter-class similarity, which puts higher requirements on the accuracy of the recognition algorithm and the ability of the algorithm to detect abnormal situations. As a result, the traditional method of elevator scene recognition based on image streams and video streams has large errors in action scene recognition, and cannot obtain accurate recognition results. It can only recognize specific abnormal scenes, and cannot achieve full scene recognition and monitoring. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide an elevator scene recognition method, device, electronic device and storage medium to solve the problem that the prior art cannot obtain accurate recognition effect, can only recognize specific abnormal scenes, and cannot achieve full scene recognition and monitoring.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for elevator scene recognition is provided, including: obtaining a video stream collected by an elevator camera, segmenting the video stream according to a preset time length, and obtaining at least one video segment; inputting the video segment into a pre-constructed elevator scene recognition model, and extracting features of the video segment using a motion recognition deep network in the elevator scene recognition model, wherein the motion recognition deep network includes three feature extraction channels; when extracting features of the video segment using the motion recognition deep network, extracting spatial dimension features using a first feature extraction channel to obtain a first feature map, extracting temporal dimension features using a second feature extraction channel to obtain a second feature map, and fusing spatial dimension features and temporal dimension features using a third feature extraction channel to obtain a third feature map; performing global average pooling on the first feature map, the second feature map, and the third feature map, and performing feature splicing on the feature maps after the global average pooling, predicting the feature maps obtained after the feature splicing, and determining the elevator scene corresponding to the video segment according to the prediction result.

[0006] According to a second aspect of an embodiment of the present disclosure, an elevator scene recognition device is provided, including: an acquisition module, configured to acquire a video stream collected by an elevator camera, segment the video stream according to a preset time length, and obtain at least one video segment; an input module, configured to input the video segment into a pre-built elevator scene recognition model, and extract features of the video segment using a motion recognition deep network in the elevator scene recognition model, wherein the motion recognition deep network contains three feature extraction channels; an extraction module, configured to extract spatial dimension features using a first feature extraction channel to obtain a first feature map, extract temporal dimension features using a second feature extraction channel to obtain a second feature map, and fuse spatial dimension features and temporal dimension features using a third feature extraction channel to obtain a third feature map when extracting features of the video segment using the motion recognition deep network; a prediction module, configured to perform global average pooling on the first feature map, the second feature map, and the third feature map, and perform feature splicing on the feature map after the global average pooling, predict the feature map obtained after the feature splicing, and determine the elevator scene corresponding to the video segment according to the prediction result.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] At least one of the above technical solutions adopted in the embodiments of the present disclosure can achieve the following beneficial effects:

[0010] By obtaining a video stream collected by an elevator camera, the video stream is segmented and processed according to a preset time length to obtain at least one video clip; the video clip is input into a pre-constructed elevator scene recognition model, and the action recognition deep network in the elevator scene recognition model is used to extract features of the video clip, wherein the action recognition deep network contains three feature extraction channels; when the action recognition deep network is used to extract features of the video clip, the first feature extraction channel is used to extract the spatial dimension features to obtain the first feature map, the second feature extraction channel is used to extract the time dimension features to obtain the second feature map, and the third feature extraction channel is used to fuse the spatial dimension features and the time dimension features to obtain the third feature map; the first feature map, the second feature map and the third feature map are globally averaged pooled, and the feature map after global average pooling is feature spliced, and the feature map obtained after feature splicing is predicted, and the elevator scene corresponding to the video clip is determined according to the prediction result. The present disclosure can reduce the recognition error of the action scene, obtain accurate elevator scene recognition effect, can recognize complex abnormal scenes, and realize elevator full scene recognition and monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 is a flow chart of an elevator scene recognition method provided by an embodiment of the present disclosure;

[0013] Figure 2 is a schematic diagram of the structure of the action recognition deep network provided by the embodiment of the present disclosure;

[0014] Figure 3 is a schematic diagram of the structure of a dual attention module provided by an embodiment of the present disclosure;

[0015] Figure 4 is a structural schematic diagram of an elevator scene recognition device provided by an embodiment of the present disclosure;

[0016] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0018] The elevator scene recognition task refers to real-time analysis of the video stream captured by the elevator camera, identification of the elevator operation status, and timely response to abnormal situations in the elevator. In the automatic recognition and monitoring tasks of elevator scenes, the elevator scenes and passenger actions that need to be identified have high information complexity, intra-class diversity, and inter-class similarity. Therefore, the traditional elevator scene recognition solution based on artificial intelligence algorithms cannot accurately identify different elevator scenes and cannot detect abnormal situations in time. Therefore, there is an urgent need for an elevator scene recognition solution that can solve the current difficulties.

[0019] In traditional elevator scene recognition algorithms, the following two recognition methods are usually used: one is a recognition algorithm based on image streams, and the other is a recognition algorithm based on video streams. Although the recognition algorithm based on image streams can automatically identify some specific objects, such as electric vehicles, passengers, pets, etc., this recognition method cannot accurately identify action scenes; and although the recognition algorithm based on video streams can identify specific action scenes through action recognition algorithms, it is difficult to obtain accurate recognition results because the algorithm design does not fully consider the complex intra-class diversity and inter-class similarity of elevator scenes and passenger actions. In addition, the above two recognition methods for elevator scenes can only identify specific abnormal scenes, that is, they can only identify single scenes, and cannot identify complex scenes, so it is impossible to achieve full-scene recognition and monitoring of elevator operation.

[0020] In view of the above problems in the prior art, the present disclosure proposes a new elevator scene recognition method based on a motion recognition deep network. The motion recognition deep network of the present disclosure mainly includes three different feature extraction channels, which are used to extract spatial dimension features, temporal dimension features and spatiotemporal fusion features respectively, so that the elevator scene recognition model can more accurately obtain the deep features of time and space in the video clips, so that the model pays more attention to important time frames and channel features, and improves the recognition effect of the model; and the embodiment of the present disclosure also labels the training data according to the category labeling rules of the whole scene, so that the trained recognition model can more accurately identify different elevator scenes, reduce the influence of the intra-class diversity and inter-class similarity of the scene on the recognition effect, and the present disclosure can accurately identify a variety of normal and abnormal scenes that may appear in the elevator, and realize the whole scene recognition and monitoring of the elevator.

[0021] Figure 1 It is a flow chart of the elevator scene recognition method provided by an embodiment of the present disclosure. Figure 1 The elevator scene recognition method can be executed by the server. Figure 1 As shown, the elevator scene recognition method may specifically include:

[0022] S101, obtaining a video stream collected by an elevator camera, and segmenting the video stream according to a preset time length to obtain at least one video segment;

[0023] S102, inputting the video clip into a pre-built elevator scene recognition model, and extracting features from the video clip using a motion recognition deep network in the elevator scene recognition model, wherein the motion recognition deep network includes three feature extraction channels;

[0024] S103, when extracting features from the video clip using the action recognition deep network, extracting spatial dimension features using a first feature extraction channel to obtain a first feature map, extracting temporal dimension features using a second feature extraction channel to obtain a second feature map, and fusing spatial dimension features and temporal dimension features using a third feature extraction channel to obtain a third feature map;

[0025] S104, performing global average pooling on the first feature map, the second feature map, and the third feature map, performing feature splicing on the feature maps after the global average pooling, predicting the feature maps obtained after the feature splicing, and determining the elevator scene corresponding to the video clip according to the prediction result.

[0026] Specifically, the video stream of the embodiment of the present disclosure refers to the video data collected by the camera installed in the elevator, and the video stream collected within a period of time is segmented to obtain multiple small video clips, and these video clips are used as the input of the elevator scene recognition model. The elevator scene recognition model is used to identify the scene of the video clip, so as to achieve the purpose of monitoring abnormal conditions in the elevator. In the following embodiments of the present disclosure, the video stream can also be referred to as video stream data or video data, and the replacement of the nouns does not constitute a limitation on the technical solution of the present disclosure.

[0027] Furthermore, the elevator scene recognition model of the embodiment of the present disclosure mainly includes a three-channel action recognition deep network based on time and channel attention. The action recognition deep network includes three feature extraction channels, each of which is used to extract features of different dimensions or fuse features. For example, the first feature extraction channel is mainly used to extract spatial dimension information, the second feature extraction channel is mainly used to extract time dimension information, and the third feature extraction channel fuses the two types of information. The action recognition deep network of the embodiment of the present disclosure is mainly used to recognize objects in video clips and 3D actions of objects, so the action recognition deep network can also be considered as a 3D action recognition deep network.

[0028] According to the technical solution provided by the embodiment of the present disclosure, the present disclosure obtains a video clip obtained by segmenting the video stream according to a preset time length, inputs the video clip into the elevator scene recognition model, and uses the action recognition deep network to extract features of the video clip; uses the first feature extraction channel to extract the spatial dimension feature to obtain the first feature map, uses the second feature extraction channel to extract the time dimension feature to obtain the second feature map, and uses the third feature extraction channel to fuse the spatial dimension feature and the time dimension feature to obtain the third feature map; performs global average pooling and feature splicing on the first feature map, the second feature map and the third feature map, predicts the feature map obtained after feature splicing, and determines the elevator scene corresponding to the video clip according to the prediction result. The present disclosure can reduce the recognition error of the action scene, obtain accurate elevator scene recognition effect, can recognize complex abnormal scenes, and realize elevator full scene recognition and monitoring.

[0029] In some embodiments, an elevator scene recognition model is constructed in the following manner, including: obtaining original video stream data for model training, dividing the original video stream data into a plurality of video segments according to a preset time length; annotating the video segments using pre-configured elevator scene category annotation rules to obtain annotated video segments, and constructing a training set based on the annotated video segments; and training the elevator scene recognition model using the training set to obtain a trained elevator scene recognition model.

[0030] Specifically, before training the elevator scene recognition model, it is necessary to first construct a data set for model training. The embodiment of the present disclosure labels the video clips in the original video stream data by designing an elevator scene category labeling rule covering the entire scene, thereby obtaining labeled data (i.e., training data) for model training.

[0031] Furthermore, after obtaining the original video stream data for model training, the original video stream data needs to be divided into video segments of a certain time interval for subsequent category labeling and model training. In practical applications, it is necessary to determine a reasonable time length of the video segment, because when the video segment is too long, multiple scenes may be included in one video segment; and if the video segment is too short, the same action scene may be separated, resulting in the algorithm failing to capture the action characteristics. Therefore, considering the individuality and integrity of the action scene, the embodiment of the present disclosure sets the duration of the video segment to 3s, that is, the entire video stream is divided into video segments of 3s in length, and each video segment is labeled with a category.

[0032] In some embodiments, the elevator scene category labeling rules include category numbers, category names and category descriptions, and the video clips are labeled using pre-configured elevator scene category labeling rules, including: analyzing the video clips to obtain description information of the elevator scene, matching the description information of the elevator scene with the category description, and determining the category description and category name corresponding to the elevator scene; labeling the video clips according to the scene priority corresponding to the category description and category name in the elevator scene category labeling rules to determine the category number of the elevator scene corresponding to each video clip.

[0033] Specifically, when the segmented video clips are annotated with scene categories, considering that the elevator scene categories have a high diversity and there is a certain overlap between different elevator scenes (such as the scene of the elevator door opening may appear at the same time as the electric vehicle scene), a sophisticated design is required to construct a suitable training set. In view of the diversity of elevator scene categories, the embodiment of the present disclosure sets an accurate and unambiguous category description for each scene category, so that there is a clear division between scene categories. The contents of the elevator scene category annotation rules provided by the embodiment of the present disclosure are described in detail below in conjunction with Table 1, which may specifically include the following contents:

[0034] Table 1 Elevator scene category labeling rules

[0035]

[0036]

[0037]

[0038] Furthermore, in view of the overlap of elevator scene categories, the video clips can be labeled with scene categories according to the principles of dynamic scene priority and simple scene priority. As shown in Table 1, the embodiment of the present disclosure divides the elevator scenes into 18 categories, and the earlier the category number, the higher the priority of the scene category labeling. Through the elevator scene category labeling rules of the embodiment of the present disclosure, the elevator scene situation at any time can be divided into one of the above 18 categories, and there is no overlap between the categories. In addition, in actual applications, the category priority in the elevator scene category labeling rules can be adjusted according to needs, such as swapping the priority of pets and large objects in the elevator.

[0039] In some embodiments, the action recognition deep network in the elevator scene recognition model is used to extract features from video clips, including: processing the video clips to obtain a preset number of image frames, using the image frames, the length, width and channel dimensions of the image frames as inputs of three feature extraction channels in the action recognition deep network, and using the three feature extraction channels to extract features of different dimensions; wherein each feature extraction channel contains the same number of convolution modules, each convolution module contains multiple three-layer residual convolution modules, the time dimension convolution size corresponding to the first layer of residual convolution in each three-layer residual convolution module is 3, and the time dimension convolution size corresponding to the second layer of residual convolution and the third layer of residual convolution is 1.

[0040] Specifically, the key to action scene recognition in elevators is to extract scene object information and action sequence information at the same time. Therefore, the disclosed embodiment proposes a three-channel 3D action recognition deep network based on time and channel attention. The network model mainly includes two aspects: first, the 3D action recognition deep network contains three feature extraction channels, the first feature extraction channel is mainly used to extract spatial dimension information, the second feature extraction channel is mainly used to extract time dimension information, and the third feature extraction channel is used to fuse the information of time and space dimensions; second, a dual attention module of time and channel is set in the convolution module (C1~C5) in each feature extraction channel. The contents of the three feature extraction channels and the feature extraction process in the action recognition deep network are described in detail below in conjunction with the accompanying drawings and specific embodiments. Figure 2 Schematic diagram of the structure of the action recognition deep network provided by the embodiment of the present disclosure. Figure 2 As shown, the action recognition deep network may include the following:

[0041] The three-channel 3D action recognition deep network based on time and channel attention contains three-way features (corresponding to three feature extraction channels). The first-way feature (i.e., the first feature extraction channel, corresponding to Figure 2The left branch in the figure mainly extracts spatial dimension information, and the second feature (i.e., the second feature extraction channel, corresponding to Figure 2 The right branch in the main branch is the time dimension information, and the third feature (i.e., the third feature extraction channel, corresponding to Figure 2 The middle branch in the 2D ResNet model is mainly used to fuse the spatial dimension information with the temporal dimension information. It should be noted that in the entire action recognition deep network, the calculation of the length, width and channel dimensions of the image frame is roughly the same as that of the 2D ResNet model. The main difference lies in the calculation of the temporal dimension.

[0042] In some embodiments, the preset number of image frames is 64 image frames, and the spatial dimension features are extracted using the first feature extraction channel to obtain a first feature map, including: sampling the 64 image frames at equal intervals according to a predetermined step size to obtain 4 image frames, using a 3D convolution with a time dimension convolution size of 1 in the first convolution module in the first feature extraction channel for calculation, and setting the time dimension size of all convolutions to 1 in the second convolution module and the third convolution module, setting the time dimension convolution size of the first layer of residual convolution to 3 in the fourth convolution module and the fifth convolution module, and setting the time dimension convolution size of the second layer of residual convolution and the third layer of residual convolution to 1; using the first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module in the first feature extraction channel to extract the spatial dimension features in the image frames in turn to obtain a first feature map.

[0043] Specifically, in the spatial dimension feature extraction of the first feature extraction channel, first, a video clip containing 64 frames of pictures is extracted at equal intervals of 4 frames with a step size of 16 frames. In the C1 stage, the 2D convolution is directly replaced with a 3D convolution with a time dimension convolution size of 1 for calculation. In the C2 and C3 stages, the time dimension size of all convolutions is set to 1. The purpose of this operation is to pay more attention to the extraction of spatial dimension information and ignore the time dimension information. In the C4 and C5 stages of the first feature extraction channel, the time dimension convolution size of the first layer of residual convolution in each block (i.e., the three-layer residual convolution module) is 3, and the time dimension convolution size of the remaining two layers of residual convolution is 1.

[0044] In some embodiments, the second feature extraction channel and the third feature extraction channel include 5 convolution modules, each convolution module has multiple three-layer residual convolution modules, the output of each layer of convolution modules in the first feature extraction channel and the second feature extraction channel is used as the input of each layer of convolution module in the third feature extraction channel, and the third feature extraction channel is used to perform convolution calculation and feature splicing to obtain a third feature map.

[0045] Specifically, in the second feature extraction channel and the third feature extraction channel, the corresponding video frames are extracted respectively, and the time dimension convolution size of the first layer of residual convolution of each block is set to 3 in stages C1 to C5, and the time dimension convolution size of the remaining two layers of residual convolution is set to 1. The features of the left and right branches are fused with the features of the middle branch through 1x1x1 convolution and feature concatenation in each stage from C1 to C5. Finally, the three-way features are globally pooled in the spatiotemporal dimension, and then the final prediction result is obtained through feature concatenation and full connection layer.

[0046] In some embodiments, the method also includes: when using the action recognition deep network to extract features from the video clip, using the dual attention module set in the convolution module of each feature extraction channel to perform a weighted operation on the time dimension features, and extracting the channel dimension features in the video clip, and weighting the features obtained after weighting the time attention on the channel dimension features; wherein the dual attention module includes time attention and channel attention, and the dual attention module is set behind the first layer of residual convolution in each convolution module.

[0047] Specifically, in addition to using three feature extraction channels to perform feature extraction and feature fusion in the spatial dimension and the temporal dimension respectively in the action recognition deep network, in order to obtain more important temporal dimension features and channel dimension features while performing feature extraction, the disclosed embodiment arranges some dual attention modules of time and channel in the action recognition deep network. The content and principle of the dual attention module are described in detail below in conjunction with the accompanying drawings and specific embodiments. Figure 3 Schematic diagram of the structure of the dual attention module provided by the embodiment of the present disclosure. Figure 3 As shown, the dual attention module may include the following:

[0048] The dual attention module contains two branches: time attention and channel attention. Each branch contains a global average pooling layer GAP, two fully connected layers FC, and a normalization layer Sigmoid. Time attention preserves the time dimension in the global average pooling layer and weights the original features in the time dimension; while channel attention preserves the channel dimension in the global average pooling layer and re-weights the features weighted by time attention in the channel dimension. The dual attention module is inserted after each block with a time dimension size of 3 in the first convolution layer of the convolution module in the above three feature extraction channels.

[0049] The disclosed embodiment sets a dual attention module of time attention and channel attention in the three channels of the action recognition deep network, so that the entire elevator scene recognition model can pay more attention to important time frames and important channel dimension information, thereby further improving the model's ability to discriminate features and enhancing the model's recognition effect.

[0050] Furthermore, the elevator scene recognition model provided by the embodiment of the present disclosure is trained using the labeled data (i.e., training data) obtained in the aforementioned embodiment. After the training of the elevator scene recognition model is completed, in actual elevator scene recognition, the video clip captured from the acquired video stream is input into the trained elevator scene recognition model for prediction, and the model can be used to output the prediction result of the scene category corresponding to the video clip.

[0051] According to the technical solution provided by the embodiment of the present disclosure, the present disclosure proposes a new elevator scene recognition method based on a motion recognition deep network. The motion recognition deep network of the present disclosure mainly includes three different feature extraction channels, which are used to extract spatial dimension features, temporal dimension features and spatiotemporal fusion features respectively, so that the elevator scene recognition model can more accurately obtain the deep features of time and space in the video clips, so that the model pays more attention to important time frames and channel features, and improves the recognition effect of the model; and the embodiment of the present disclosure also labels the training data through the category labeling rules of the whole scene, so that the trained recognition model can more accurately identify different elevator scenes, reduce the influence of the intra-class diversity and inter-class similarity of the scene on the recognition effect, and the present disclosure can accurately identify a variety of normal and abnormal scenes that may appear in the elevator, and realize the full-scene recognition and monitoring of the elevator.

[0052] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0053] Figure 4 Schematic diagram of the structure of the elevator scene recognition device provided by the embodiment of the present disclosure. Figure 4 As shown, the elevator scene recognition device includes:

[0054] The acquisition module 401 is configured to acquire a video stream collected by an elevator camera, and segment the video stream according to a preset time length to obtain at least one video segment;

[0055] An input module 402 is configured to input the video clip into a pre-built elevator scene recognition model, and extract features from the video clip using a motion recognition deep network in the elevator scene recognition model, wherein the motion recognition deep network includes three feature extraction channels;

[0056] The extraction module 403 is configured to extract the spatial dimension features using the first feature extraction channel to obtain a first feature map, extract the temporal dimension features using the second feature extraction channel to obtain a second feature map, and fuse the spatial dimension features and the temporal dimension features using the third feature extraction channel to obtain a third feature map when extracting features from the video clip using the action recognition deep network;

[0057] The prediction module 404 is configured to perform global average pooling on the first feature map, the second feature map and the third feature map, perform feature splicing on the feature maps after the global average pooling, predict the feature maps obtained after the feature splicing, and determine the elevator scene corresponding to the video clip according to the prediction results.

[0058] In some embodiments, Figure 4 The construction module 405 constructs the elevator scene recognition model in the following manner, including: obtaining original video stream data for model training, dividing the original video stream data into a plurality of video segments according to a preset time length; annotating the video segments using pre-configured elevator scene category annotation rules to obtain annotated video segments, and constructing a training set based on the annotated video segments; and training the elevator scene recognition model using the training set to obtain a trained elevator scene recognition model.

[0059] In some embodiments, the elevator scene category labeling rules include category number, category name and category description. Figure 4 The construction module 405 analyzes the video clips to obtain the description information of the elevator scene, matches the description information of the elevator scene with the category description, and determines the category description and category name corresponding to the elevator scene; and labels the video clips according to the scene priority corresponding to the category description and the category name in the elevator scene category labeling rule to determine the category number of the elevator scene corresponding to each video clip.

[0060] In some embodiments, Figure 4 The input module 402 processes the video clip to obtain a preset number of image frames, and uses the image frames, the length, width and channel dimensions of the image frames as inputs of three feature extraction channels in the action recognition deep network, and uses the three feature extraction channels to extract features of different dimensions; wherein each feature extraction channel includes the same number of convolution modules, each convolution module includes multiple three-layer residual convolution modules, the time dimension convolution size corresponding to the first layer of residual convolution in each three-layer residual convolution module is 3, and the time dimension convolution size corresponding to the second layer of residual convolution and the third layer of residual convolution is 1.

[0061] In some embodiments, the preset number of image frames is 64 image frames. Figure 4The extraction module 403 extracts 64 image frames at equal intervals according to a predetermined step size to obtain 4 image frames, and uses 3D convolution with a time dimension convolution size of 1 to perform calculations in the first convolution module in the first feature extraction channel, and sets the time dimension sizes of all convolutions to 1 in the second convolution module and the third convolution module, and sets the time dimension convolution size of the first layer of residual convolution to 3 in the fourth convolution module and the fifth convolution module, and sets the time dimension convolution size of the second layer of residual convolution and the third layer of residual convolution to 1; uses the first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module in the first feature extraction channel to extract spatial dimension features in the image frame in turn to obtain a first feature map.

[0062] In some embodiments, the second feature extraction channel and the third feature extraction channel include 5 convolution modules, each convolution module has multiple three-layer residual convolution modules, the output of each layer of convolution modules in the first feature extraction channel and the second feature extraction channel is used as the input of each layer of convolution module in the third feature extraction channel, and the third feature extraction channel is used to perform convolution calculation and feature splicing to obtain a third feature map.

[0063] In some embodiments, Figure 4 When the extraction module 403 uses the action recognition deep network to extract features from the video clip, it uses the dual attention module set in the convolution module of each feature extraction channel to perform a weighted operation on the time dimension features, and extracts the channel dimension features in the video clip, and weights the features obtained after weighting the time attention on the channel dimension features; wherein the dual attention module includes time attention and channel attention, and the dual attention module is set behind the first layer of residual convolution in each convolution module.

[0064] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0065] Figure 5 Schematic diagram of the structure of the electronic device 5 provided by the embodiment of the present disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0066] Exemplarily, the computer program 503 may be divided into one or more modules / units, which are stored in the memory 502 and executed by the processor 501 to complete the present disclosure. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 503 in the electronic device 5.

[0067] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 It is only an example of the electronic device 5 and does not constitute a limitation of the electronic device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0068] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0069] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. Further, the memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device. The memory 502 may also be used to temporarily store data that has been output or is to be output.

[0070] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0071] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0072] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0073] In the embodiments provided in the present disclosure, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which may be electrical, mechanical or other forms.

[0074] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0076] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0077] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. An elevator scene recognition method, characterized in that: include: Acquire a video stream captured by an elevator camera, and segment the video stream according to a preset time length to obtain at least one video segment; Input the video clip into a pre-built elevator scene recognition model, and use a motion recognition deep network in the elevator scene recognition model to extract features from the video clip, wherein the motion recognition deep network includes three feature extraction channels; When extracting features from the video clip using the action recognition deep network, a first feature extraction channel is used to extract spatial dimension features to obtain a first feature map, a second feature extraction channel is used to extract temporal dimension features to obtain a second feature map, and a third feature extraction channel is used to fuse the spatial dimension features and the temporal dimension features to obtain a third feature map; Performing global average pooling on the first feature map, the second feature map, and the third feature map, and performing feature splicing on the feature maps after the global average pooling, predicting the feature maps obtained after the feature splicing, and determining the elevator scene corresponding to the video clip according to the prediction result; The step of extracting features from the video clip using the action recognition deep network in the elevator scene recognition model includes: Processing the video clips to obtain a preset number of image frames, using the image frames, the length, width, and channel dimensions of the image frames as inputs of three feature extraction channels in the action recognition deep network, and using the three feature extraction channels to extract features of different dimensions; Wherein, each feature extraction channel includes the same number of convolution modules, and each of the convolution modules includes multiple three-layer residual convolution modules; The output of each layer of convolution modules in the first feature extraction channel and the second feature extraction channel is used as the input of the corresponding layer of convolution modules in the third feature extraction channel, and the third feature extraction channel is used to perform convolution calculation and feature splicing to obtain a third feature map.

2. The method according to claim 1, characterized in that The elevator scene recognition model is constructed in the following manner, including: Obtaining original video stream data for model training, and dividing the original video stream data into a plurality of video segments according to a preset time length; The video clips are labeled using pre-configured elevator scene category labeling rules to obtain labeled video clips, and a training set is constructed based on the labeled video clips; The elevator scene recognition model is trained using the training set to obtain a trained elevator scene recognition model.

3. The method according to claim 2, characterized in that The elevator scene category labeling rule includes a category number, a category name and a category description, and labeling the video clip using the pre-configured elevator scene category labeling rule includes: Analyze the video clip to obtain description information of the elevator scene, match the description information of the elevator scene with the category description, and determine the category description and category name corresponding to the elevator scene; The video clips are labeled according to the category description in the elevator scene category labeling rule and the scene priority corresponding to the category name to determine the category number of the elevator scene corresponding to each of the video clips.

4. The method according to claim 1, characterized in that: The time dimension convolution size corresponding to the first layer of residual convolution in each of the three-layer residual convolution modules is 3, and the time dimension convolution size corresponding to the second layer of residual convolution and the third layer of residual convolution is 1.

5. The method according to claim 4, characterized in that The preset number of image frames is 64 image frames, and the first feature extraction channel is used to extract the spatial dimension feature to obtain the first feature map, including: 64 image frames are equally spaced according to a predetermined step size to obtain 4 image frames, 3D convolution with a time dimension convolution size of 1 is used for calculation in the first convolution module in the first feature extraction channel, and the time dimension sizes of all convolutions are set to 1 in the second convolution module and the third convolution module, and the time dimension convolution size of the first layer of residual convolution is set to 3 in the fourth convolution module and the fifth convolution module, and the time dimension convolution size of the second layer of residual convolution and the third layer of residual convolution is set to 1; The first convolution module, the second convolution module, the third convolution module, the fourth convolution module and the fifth convolution module in the first feature extraction channel are used to extract the spatial dimension features in the image frame in sequence to obtain a first feature map.

6. The method according to claim 4, characterized in that The second feature extraction channel and the third feature extraction channel include 5 convolution modules, and each of the convolution modules has multiple three-layer residual convolution modules.

7. The method according to claim 1, characterized in that The method further comprises: When extracting features from the video clip using the action recognition deep network, the original features are weighted in the time dimension using the time attention in the dual attention module set in the convolution module of each feature extraction channel, and channel dimension features in the video clip are extracted, and features obtained after weighting the time attention are weighted on the channel dimension features; Among them, the dual attention module includes time attention and channel attention, and the dual attention module is arranged behind the first layer of residual convolution in each of the convolution modules.

8. An elevator scene recognition device, characterized in that: include: An acquisition module is configured to acquire a video stream collected by an elevator camera, and segment the video stream according to a preset time length to obtain at least one video segment; An input module is configured to input the video clip into a pre-built elevator scene recognition model, and extract features from the video clip using a motion recognition deep network in the elevator scene recognition model, wherein the motion recognition deep network includes three feature extraction channels; The extraction module is configured to, when extracting features of the video clip using the action recognition deep network, extract spatial dimension features using a first feature extraction channel to obtain a first feature map, extract temporal dimension features using a second feature extraction channel to obtain a second feature map, and fuse the spatial dimension features and the temporal dimension features using a third feature extraction channel to obtain a third feature map; A prediction module is configured to perform global average pooling on the first feature map, the second feature map, and the third feature map, perform feature splicing on the feature maps after the global average pooling, predict the feature maps obtained after the feature splicing, and determine the elevator scene corresponding to the video clip according to the prediction result; Among them, the input module is used to process the video clip to obtain a preset number of image frames, and use the image frames, the length, width and channel dimensions of the image frames as inputs of three feature extraction channels in the action recognition deep network, and use the three feature extraction channels to extract features of different dimensions; wherein each feature extraction channel contains the same number of convolution modules, and each convolution module contains multiple three-layer residual convolution modules; the output of each layer of convolution modules in the first feature extraction channel and the second feature extraction channel is used as the input of the corresponding layer of convolution modules in the third feature extraction channel, and the third feature extraction channel is used to perform convolution calculation and feature splicing to obtain a third feature map.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Three-branch network behavior identification method based on multipath space-time feature enhanced fusion

    CN111709351A

  • Dangerous behavior detection method and system based on space-time double-flow convolutional neural network

    CN113239822A

  • Behavior recognition method and system

    CN113378600A

  • Video processing method and apparatus, and electronic device and storage medium

    US20210326597A1