Abnormality detection method and device, equipment, medium and program product
By combining identity recognition and action recognition, calculating the frame extraction ratio based on importance weights, and employing multi-scale feature fusion and attention operations, the accuracy and real-time performance issues of anomaly detection in video surveillance are resolved, thereby improving recognition efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-10
AI Technical Summary
Existing technologies struggle to balance accuracy and real-time performance in anomaly detection during video surveillance. High frame rate leads to slower data processing, while low frame rate may miss important information. Action recognition accuracy is insufficient, and misjudgments or omissions are prone to occur, especially in complex backgrounds.
The identity of the person is initially determined by identity recognition. The frame sampling ratio is calculated according to the importance weight. The action recognition model is combined for fine detection. Multi-scale feature fusion and attention operation are adopted to improve the recognition efficiency and accuracy.
It achieves a balance between the accuracy and real-time performance of anomaly detection while ensuring detection precision, improves the accuracy of action recognition in complex backgrounds, and reduces the consumption of computing resources.
Smart Images
Figure CN121838014A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology and can be used in the financial technology field, specifically to an anomaly detection method, device, equipment, medium, and program product. Background Technology
[0002] In the daily operations of financial institutions, it is necessary to monitor sensitive areas in real time through video surveillance and identify abnormal targets in the video surveillance to ensure the security of the financial institutions' operations. For example, when abnormal personnel suddenly appear in a sensitive area, or when personnel suddenly perform some abnormal behavior, it is necessary to issue timely and accurate warnings.
[0003] To accurately identify abnormal targets or their unusual behavior in videos, related technologies typically extract multiple frames from continuous video data at a preset frame extraction ratio, and then use deep learning models to process and identify the extracted frames. However, this approach struggles to balance accuracy and real-time performance in anomaly detection. While using a higher frame extraction ratio can improve accuracy, the increased data processing volume slows down the process, reducing real-time performance. Conversely, using a lower frame extraction ratio may miss important information, resulting in insufficient accuracy in anomaly detection. Summary of the Invention
[0004] In view of the above problems, this application provides anomaly detection methods, apparatus, equipment, media and program products that can balance the accuracy and real-time performance of anomaly detection.
[0005] According to a first aspect of this application, an anomaly detection method is provided, comprising: acquiring surveillance video data of a designated area; identifying the identity features of a person in a first target frame image in the surveillance video data; determining an importance weight based on the identity features and the regional features of the designated area; calculating a frame extraction ratio of the surveillance video data based on the product of a preset ratio parameter and the importance weight; extracting N second target frame images from the surveillance video data based on the frame extraction ratio; wherein N is an integer greater than 1; extracting feature data from the N second target frame images; identifying the feature data to obtain a person's action recognition result; and determining an anomaly detection result based on the matching relationship between the designated area, the identity features, and the action recognition result.
[0006] According to an embodiment of this application, the identity features include the person's identity recognition result and the person's clothing recognition result. The importance weight is determined based on the identity features and the regional features of a set area, including: determining a first weight based on the identity recognition result; determining a second weight based on the matching relationship between the identity recognition result and the regional features; determining a third weight based on the matching relationship between the identity recognition result and the clothing recognition result; and determining the importance weight based on the first weight, the second weight, the third weight, and the regional weight corresponding to the regional features.
[0007] According to an embodiment of this application, identifying the identity features of a person in a first target frame image includes: inputting the first target frame image into a shallow feature extraction module to obtain a first feature; inputting the first feature into M concatenated basic feature extraction modules to obtain a second feature; wherein M is an integer greater than 1; superimposing the first feature and the second feature to obtain a third feature; and inputting the third feature into an identity recognition module to obtain an identity feature; wherein the basic feature extraction module includes multiple parallel feature extraction channels, and each feature extraction channel includes a downsampling module, a depthwise convolution module, and an upsampling module connected in sequence, wherein the downsampling module and the upsampling module in each feature extraction channel have the same sampling factor, and the downsampling module and the upsampling module between different feature extraction channels have different sampling factors.
[0008] According to an embodiment of this application, the basic feature extraction module further includes a convolutional channel mixing module, which includes a first convolutional module, a nonlinear mapping module, and a second convolutional module connected in sequence, wherein the convolutional kernel sizes of the first convolutional module and the second convolutional module are different.
[0009] According to an embodiment of this application, feature data of N second target frame images are extracted, and the feature data is used to identify the action recognition result of the person. The process includes: for each second target frame image, performing multiple attention operations sequentially to obtain multi-level attention features of the second target frame image and intermediate attention features of each attention operation; fusing the multi-level attention features corresponding to each second target frame image to obtain a first fusion feature; fusing the intermediate attention features corresponding to each second target frame image and the first fusion feature to obtain a second fusion feature; inputting the first fusion feature into a multi-scale feature fusion module to obtain a first multi-scale fusion feature; the multi-scale feature fusion module is used to obtain local and global features of the input features; inputting the second fusion feature into the multi-scale feature fusion module to obtain a second multi-scale fusion feature; fusing the first multi-scale fusion feature and the second multi-scale fusion feature to obtain a third multi-scale fusion feature; and inputting the third multi-scale fusion feature into an action recognition module to obtain an action recognition result.
[0010] According to an embodiment of this application, the multi-scale feature fusion module includes a segmentation module, a pooling module, and a fusion module. The segmentation module is used to segment the input features of the multi-scale feature fusion module according to different ratios to obtain multiple segmented features. The pooling module is used to perform global average pooling and global max pooling operations on each segmented feature respectively. The fusion module is used to superimpose the output features of the pooling module to generate the output features of the multi-scale feature fusion module.
[0011] According to an embodiment of this application, the first fused feature is obtained by fusing the multi-level attention features corresponding to each second target frame image, including: splicing the multi-level attention features to obtain a first spliced feature; performing max pooling, mean pooling, and mode pooling operations on the spliced features respectively to obtain corresponding max pooling features, mean pooling features, and mode pooling features; splicing the max pooling features, mean pooling features, and mode pooling features to obtain a second spliced feature; and aggregating the second spliced feature to obtain the first fused feature.
[0012] According to a second aspect of this application, an anomaly detection device is provided, comprising: a data acquisition module for acquiring surveillance video data of a set area; an identity recognition module for identifying the identity features of a person in a first target frame image in the surveillance video data; a calculation module for determining an importance weight based on the identity features and the regional features of the set area, and calculating a frame extraction ratio of the surveillance video data based on the product of a preset ratio parameter and the importance weight; an image extraction module for extracting N second target frame images from the surveillance video data based on the frame extraction ratio, wherein N is an integer greater than 1; an action recognition module for extracting feature data from the N second target frame images and recognizing the feature data to obtain the action recognition result of the person; and a detection module for determining an anomaly detection result based on the matching relationship between the set area, the identity features, and the action recognition result.
[0013] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.
[0014] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.
[0015] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.
[0016] The above one or more embodiments have the following beneficial effects: The anomaly detection process integrates identity recognition and action recognition. Identity recognition initially determines the identity of the person, and an importance weight is determined based on the person's identity and the characteristics of the monitored area. The frame extraction ratio is then calculated based on this importance weight. The importance weight is related to the importance of subsequent action recognition tasks. For low-importance tasks, such as identifying a person whose identity is normal, the probability of anomalies is low, allowing for a lower frame extraction ratio to save computational resources and improve real-time performance. Conversely, for high-importance tasks, such as identifying personnel who should not be in the monitored area, a higher frame extraction ratio can be used to extract frames from the video data for refined identification, thus achieving a balance between accuracy and real-time performance while ensuring detection precision. Attached Figure Description
[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:
[0018] Figure 1 The illustrations depict application scenarios of anomaly detection methods, apparatuses, devices, media, and program products according to embodiments of this application.
[0019] Figure 2 A flowchart illustrating an anomaly detection method according to an embodiment of this application is shown schematically.
[0020] Figure 3 This illustration schematically shows a structural diagram of the identity recognition model in the anomaly detection method according to an embodiment of this application;
[0021] Figure 4 This illustration schematically shows a structural diagram of the action recognition model in the anomaly detection method according to an embodiment of this application;
[0022] Figure 5 This illustration schematically shows a structural diagram of a multi-feature set pooling module in an action recognition model according to an embodiment of this application;
[0023] Figure 6 This illustration schematically shows a structural diagram of a multi-scale feature fusion module in an action recognition model according to an embodiment of this application;
[0024] Figure 7 A schematic diagram illustrating the structure of an anomaly detection device according to an embodiment of this application is shown; and
[0025] Figure 8 A block diagram schematically illustrates an electronic device suitable for an anomaly detection method according to an embodiment of this application. Detailed Implementation
[0026] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.
[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.
[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.
[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).
[0030] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.
[0031] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.
[0032] Figure 1 The illustration shows an application scenario diagram of the anomaly detection method according to an embodiment of this application.
[0033] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0034] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).
[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers. Furthermore, the first terminal device 101, the second terminal device 102, and the third terminal device 103 can also be business processing terminals with image acquisition capabilities at the business site.
[0036] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.
[0037] It should be noted that the anomaly detection method provided in this application embodiment can generally be executed by the server 105. Correspondingly, the anomaly detection device provided in this application embodiment can generally be located in the server 105. The anomaly detection method provided in this application embodiment can also be executed by a server or server cluster that is different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the anomaly detection device provided in this application embodiment can also be located in a server or server cluster that is different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105.
[0038] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0039] Existing anomaly detection solutions based on video surveillance often struggle to balance accuracy and real-time performance. While using a higher frame extraction ratio can improve the accuracy of anomaly detection, the increased data processing volume slows down the data processing and reduces real-time performance. On the other hand, using a lower frame extraction ratio may miss important information, resulting in insufficient accuracy of anomaly detection.
[0040] Furthermore, the accuracy of action recognition in related technologies needs improvement. It struggles to accurately capture subtle movements, such as in low-light conditions, when multiple people are obstructing the view, or in reflective environments. For actions with high similarity, misjudgment or missed detection can easily occur, affecting the accuracy of recognition. Additionally, to ensure accuracy, related technologies typically use a high frame sampling ratio, resulting in poor real-time performance. There is a significant delay between the occurrence of an action and the recognition alert, making it difficult to promptly detect and stop abnormal behavior. Moreover, financial institutions (such as bank branches) experience frequent personnel movement and complex, ever-changing backgrounds, making traditional deep learning-based recognition algorithms susceptible to interference and unable to reliably recognize actions.
[0041] To address at least one of the aforementioned problems, embodiments of this application provide anomaly detection methods, apparatuses, devices, media, and program products to improve anomaly detection accuracy and balance the accuracy and real-time performance of anomaly detection.
[0042] The following will be based on Figure 1 The described scene, through Figure 2 The anomaly detection method according to the embodiments of this application will be described in detail.
[0043] Figure 2A flowchart illustrating an anomaly detection method according to an embodiment of this application is shown schematically.
[0044] like Figure 2 As shown, the anomaly detection method of this embodiment includes operations S210 to S260.
[0045] When operating S210, acquire surveillance video data for the designated area.
[0046] In operation S220, for the first target frame image in the surveillance video data, the identity characteristics of the person in the first target frame image are identified.
[0047] In operation S230, the importance weight is determined based on the identity characteristics and the regional characteristics of the set area, and the frame extraction ratio of the monitoring video data is calculated based on the product of the preset ratio parameter and the importance weight.
[0048] In operation S240, N second target frame images are extracted from the monitoring video data based on the frame extraction ratio; where N is an integer greater than 1.
[0049] In operation S250, feature data of N second target frame images are extracted, and the feature data is used to identify the human action recognition result.
[0050] When operating S260, the anomaly detection result is determined based on the matching relationship between the set area, identity features, and action recognition results.
[0051] In the embodiments of this application, the anomaly detection process integrates two processes: identity recognition and action recognition. Identity recognition initially determines the identity of the person, and an importance weight is determined based on the person's identity and the characteristics of the monitored area. The frame extraction ratio is then calculated based on this importance weight. The importance weight is correlated with the importance of subsequent action recognition tasks. For low-importance tasks, such as identifying a person whose identity is normal, the probability of anomalies is low, allowing for a lower frame extraction ratio to save computational resources and improve real-time performance. Conversely, for high-importance tasks, such as identifying personnel who should not be within the monitored area, a higher frame extraction ratio can be used to extract frames from the video data for more refined identification. This achieves a balance between accuracy and real-time performance while ensuring detection precision.
[0052] In operation S220, in the embodiments of this application, the identified personnel's identity characteristics include personnel identity category and clothing category. Identity category can be used to determine whether a person is an internal staff member, and if so, to determine their specific department and permission level, in order to determine whether the person has the authority to appear in the scenario and the actions allowed to be performed in the scenario, such as accessing files, using a computer, copying files, etc. Clothing category can be used to determine whether the person's clothing meets the dress requirements of the scenario, and can also be used in conjunction with identity category to determine whether the internal staff member's attire conforms to regulations.
[0053] In the embodiments of this application, the identification of identity features is implemented based on an identity recognition model. Identifying the identity features of a person in a first target frame image includes: inputting the first target frame image into a shallow feature extraction module to obtain a first feature; inputting the first feature into M concatenated basic feature extraction modules to obtain a second feature; where M is an integer greater than 1; superimposing the first and second features to obtain a third feature; and inputting the third feature into an identity recognition module to obtain an identity feature. The basic feature extraction module includes multiple parallel feature extraction channels, each including a downsampling module, a depthwise convolution module, and an upsampling module connected in sequence. The downsampling and upsampling modules in each feature extraction channel have the same sampling factor, while the sampling factors of the downsampling and upsampling modules differ between feature extraction channels.
[0054] In the embodiments of this application, multiple cascaded basic feature extraction modules are used to extract spatial features from the input first target frame image. These modules employ parallel feature extraction channels, combined with different downsampling and upsampling ratios, to extract features at different scales. By utilizing depthwise convolution, a function similar to the transformer model is achieved without consuming excessive computational resources, thus improving recognition efficiency. Since the embodiments of this application use a combination of identity recognition and action recognition for anomaly detection, identity recognition is essentially a preliminary identification process. Its accuracy requirements are not high; the main focus is on quickly identifying potential anomalies. Then, combined with the calculation of the frame sampling ratio, refined anomaly detection is achieved based on action recognition.
[0055] In the embodiments of this application, the first target frame image can be obtained by extracting frames from the monitoring video data based on a preset frame extraction ratio, for example, extracting 1 frame per second.
[0056] Figure 3 The diagram illustrates the structure of the identity recognition model in the anomaly detection method according to an embodiment of this application. Figure 3As shown in the embodiments of this application, the identity recognition model includes firstly using a convolutional module with a kernel size of 3×3 pixels as a shallow feature extraction module to convert the input low-resolution image into a feature space and generate shallow features, i.e., the first feature. Then, the first feature is input into multiple cascaded basic feature extraction modules for feature extraction to obtain the second feature. The second feature output by the basic feature extraction modules is added to the first feature to achieve global residual connections, thereby learning high-frequency details and obtaining the third feature. After adjusting the channels of a convolutional layer, the third feature is input into an identity recognition module consisting of two fully connected layers and a softmax activation function layer for identity recognition, resulting in the identity recognition result, i.e., the identity feature.
[0057] In the embodiments of this application, the basic feature extraction module has spatial adaptive feature adjustment characteristics and rapidly reconstructs features through lightweight upsampling and downsampling layers. Compared with the traditional transformer model, it also has the ability to learn long-range dependencies, consumes less computational resources, and can learn useful features from multi-scale feature representations, allowing for better exploration of more useful features for subsequent recognition. To reduce model complexity and obtain pyramid-like feature representations, the normalized feature input is subjected to channel segmentation, generating four components. There are four feature extraction channels: the first feature extraction channel includes a depthwise convolutional module with a kernel size of 3×3; the second feature extraction channel includes a 2x downsampling module, a depthwise convolutional module with a kernel size of 3×3, and a difference upsampling module connected in sequence; the third feature extraction channel includes a 4x downsampling module, a depthwise convolutional module with a kernel size of 3×3, and a difference upsampling module connected in sequence; and the fourth feature extraction channel includes an 8x downsampling module, a depthwise convolutional module with a kernel size of 3×3, and a difference upsampling module connected in sequence. The upsampling module uses nearest-neighbor interpolation to capture pyramid-like information.
[0058] In the embodiments of this application, the basic feature extraction module further includes a convolutional channel mixing module, which includes a first convolutional module, a nonlinear mapping module, and a second convolutional module connected in sequence. The first convolutional module and the second convolutional module have different convolutional kernel sizes.
[0059] In the embodiments of this application, the basic feature extraction module further includes a convolutional channel mixing module. Using convolutions with different kernel sizes can balance the extraction of local spatial features with computational efficiency. In the embodiments of this application, the first convolutional module uses a 3×3 kernel to encode the spatial local context and double the number of input feature channels in the mixed channels. The second convolutional module uses a 1×1 kernel to reduce the number of channels to the original input dimension, thereby reducing computational cost and enhancing channel interaction. Furthermore, in the embodiments of this application, the nonlinear mapping module uses the GELU function instead of the traditional ReLU function to smooth nonlinearities and improve gradient flow, thereby enhancing training stability.
[0060] Specifically, in the embodiments of this application, the convolutional channel mixing module is placed after multiple parallel feature extraction channels. For the output features of each feature extraction channel, after concatenation using concat, they are processed by a convolutional module with a kernel size of 1×1 and a GELU activation function module, and then residually connected with the input features to obtain the input features of the convolutional channel mixing module. The convolutional channel mixing module includes a normalization layer, a convolutional module with a kernel size of 3×3, an activation function layer, and a convolutional module with a kernel size of 1×1 connected in sequence, and a residual connection is introduced to add the output features of the convolutional channel mixing module with the input features of the convolutional channel mixing module to obtain the output features of the overall basic feature extraction module. Placing the activation function layer before the convolutional module can better stabilize model training and obtain better results.
[0061] In embodiments of this application, the identity recognition module includes a first recognition module for recognizing clothing information and a second recognition module for recognizing identity information, capable of outputting the clothing category and identity category of a person to identify the person's identity and clothing.
[0062] In the embodiments of this application, the identity features include the person's identity recognition result and the person's clothing recognition result. In operation S230, the importance weight is determined based on the identity features and the regional features of the set area, including: determining a first weight based on the identity recognition result; determining a second weight based on the matching relationship between the identity recognition result and the regional features; determining a third weight based on the matching relationship between the identity recognition result and the clothing recognition result; and determining the importance weight based on the first weight, the second weight, the third weight, and the regional weight corresponding to the regional features.
[0063] In the embodiments of this application, the importance weight is calculated based on the identity recognition results and the regional features of the monitoring scene. The relationship between different personnel identities and different scene characteristics is fully considered. The frame extraction ratio is calculated, which can improve the accuracy of frame extraction and thus ensure the balance between recognition accuracy and real-time performance.
[0064] Specifically, defining the characteristics of a region can include its type and the corresponding access permissions. If the region type indicates it's an important region, such as a bank warehouse, then this region needs focused monitoring, and the corresponding frame rate should be increased. Different region weights can be preset for different region types as the basis for calculating importance weights. For example, the weight of a normal region can be set to 1, and the weight of an important region can be set to 2. Different weights can also be preset for specific regions.
[0065] Based on the identification results, a first weight can be determined to indicate the importance of the person's identity. For example, if the person is identified as an internal staff member, the first weight can be set to 1, and if the person is identified as an external staff member, the first weight can be set to 3.
[0066] Access permissions for a region can be combined with the identification results of individuals to calculate a second weight. For example, if a region's access permission is at level one, but the identification result indicates that the individual's permission is at level two (lower than level one), the two cannot match, indicating that the individual does not have access permission to that region. In this case, the second weight is set to a corresponding preset value, such as 2. If the identification result matches the region's access permissions, the second weight can be set to 1.
[0067] Identity recognition results can also be combined with clothing recognition results to calculate a third weight. For example, if a person's clothing recognition result shows that they are not wearing work clothes, but the identity recognition result indicates that the person is an internal staff member, it can be identified that the person's identity and clothing do not match, which may indicate a violation or other risk. The third weight can be set to a corresponding preset value, such as 2. If the person's identity and clothing match, the third weight can be set to 1.
[0068] The importance weight is calculated based on the first weight, the second weight, the third weight, and the regional weight corresponding to the regional feature. It can be calculated by multiplying the first weight, the second weight, the third weight, and the regional weight, or by using weighted summation or other calculation methods.
[0069] The frame extraction ratio for surveillance video data can be calculated by multiplying a preset ratio parameter (which can be an initial frame extraction ratio, such as 1 frame / second) with an importance weight. For example, if the calculated importance weight is 6, then the frame extraction ratio can be 6 frames / second.
[0070] In operation S240, based on the calculated frame extraction ratio, the surveillance video data is processed by frame extraction to obtain N second target frame images, so as to identify personnel actions through the second target frame images.
[0071] In operation S250, the action recognition result is generated based on the action recognition model. In the embodiments of this application, feature data of N second target frame images are extracted, and the action recognition result of the person is obtained by recognizing the feature data, including: for each second target frame image, multiple attention operations are performed sequentially to obtain multi-level attention features of the second target frame image and intermediate attention features of each attention operation; the multi-level attention features corresponding to each second target frame image are fused to obtain a first fused feature; the intermediate attention features corresponding to each second target frame image and the first fused feature are fused to obtain a second fused feature; the first fused feature is input into a multi-scale feature fusion module to obtain a first multi-scale fused feature; the multi-scale feature fusion module is used to obtain local features and global features of the input features; the second fused feature is input into the multi-scale feature fusion module to obtain a second multi-scale fused feature; the first multi-scale fused feature and the second multi-scale fused feature are fused to obtain a third multi-scale fused feature; the third multi-scale fused feature is input into the action recognition module to obtain the action recognition result.
[0072] In the embodiments of this application, during the multi-scale feature extraction and fusion process of multiple second frame images, intermediate attention features from different processing stages are extracted and fused, and then fused with the extracted multi-scale features, making full use of features at different scales and improving the accuracy of subsequent recognition.
[0073] Figure 4 The diagram illustrates the structure of the action recognition model in the anomaly detection method according to an embodiment of this application. Figure 4 As shown, the deep learning-based action recognition model can analyze detected human actions in real time and identify various subtle movements, such as hand, body, and head movements. By fully utilizing features at different scales, the model can accurately capture subtle changes in movements and maintain high recognition accuracy even in complex backgrounds.
[0074] For each input second target frame image, the original skeleton data of the person in the image is first extracted to facilitate subsequent action recognition. The extracted data is then optimized using self-attention blocks, further processed using a transformer module for feature extraction, and then downsampled through a pooling layer to complete the first attention operation, obtaining the first intermediate attention features. The first intermediate attention features are then fused using a multi-scale feature fusion module to obtain the first intermediate fused features.
[0075] In the second attention operation, the same operation is performed on the first intermediate fused feature and each of the first intermediate attention features to ensure consistency in feature scale and facilitate subsequent fusion. In the second attention operation, the first intermediate attention features are sequentially input into the transformer module for feature extraction and pooling layer for downsampling to obtain the second intermediate attention features. The multi-scale feature fusion module is then used to fuse the second intermediate attention features, and finally, these second intermediate attention features are added to the first intermediate fused features processed by the transformer module and pooling layer to obtain the final second intermediate attention feature.
[0076] In the third attention operation, the same operation is performed on the second intermediate fusion feature and each of the second intermediate attention features to ensure consistency of feature scale. In the embodiments of this application, the attention operation is performed three times. In the third attention operation, no downsampling operation is required; only the second intermediate attention features are input into the transformer module for processing to obtain the multi-level attention features corresponding to each second intermediate attention feature. The multi-level attention features are then input into the multi-scale feature fusion module for fusion to obtain the first fusion feature. After the second intermediate attention features are input into the transformer module for processing, they are added to the first fusion feature to obtain the second fusion feature.
[0077] The first fused feature is input into the multi-scale feature fusion module to obtain the first multi-scale fused feature. The second fused feature is then input into the same module to obtain the second multi-scale fused feature. Subsequently, the first and second multi-scale fused features are processed by a fully connected layer and then concatenated using a cat-like algorithm along the channel direction to obtain the third multi-scale fused feature. Finally, the third multi-scale fused feature is input into the action recognition module to obtain the action recognition result. The action recognition module includes a softmax activation function layer, which can input the classification result for the action category.
[0078] In the embodiments of this application, the multi-scale feature fusion module includes a segmentation module, a pooling module, and a fusion module. The segmentation module is used to segment the input features of the multi-scale feature fusion module according to different ratios to obtain multiple segmented features. The pooling module is used to perform global average pooling and global max pooling operations on each segmented feature respectively. The fusion module is used to superimpose the output features of the pooling module to generate the output features of the multi-scale feature fusion module.
[0079] In the embodiments of this application, the multi-scale feature fusion module segments the input features into features of different scales and performs global average pooling and global max pooling operations on them respectively. This fully extracts multi-scale features, making the features both local and global, thus improving the accuracy of action recognition. Furthermore, in the embodiments of this application, the multi-scale feature fusion module in the action recognition model is followed by a fully connected layer, which further improves the discriminative power of the features.
[0080] Figure 6 The diagram illustrates the structure of a multi-scale feature fusion module in an action recognition model according to an embodiment of this application. Figure 6 As shown in the embodiments of this application, the multi-scale feature fusion module divides the input features into 1, 2, 4, and 8 parts respectively, obtaining 15 features. It then performs global average pooling and global max pooling operations on the segmented features respectively. Finally, the features obtained by the global average pooling and global max pooling operations are superimposed, for example, by using add superposition to obtain the output features.
[0081] In the embodiments of this application, the multi-level attention features corresponding to each second target frame image are fused to obtain a first fused feature, including: splicing the multi-level attention features to obtain a first spliced feature; performing max pooling, mean pooling, and mode pooling operations on the spliced features respectively to obtain corresponding max pooling features, mean pooling features, and mode pooling features; splicing the max pooling features, mean pooling features, and mode pooling features to obtain a second spliced feature; and aggregating the second spliced feature to obtain the first fused feature.
[0082] In the embodiments of this application, multiple pooling methods are used in the feature fusion process. On the one hand, key features can be preserved and features at different scales can be fully extracted. On the other hand, compared with the simple use of cat connection for channel stacking fusion, the pooling method can reduce the amount of computation and improve the real-time performance of recognition.
[0083] Figure 5 The diagram illustrates the structure of a multi-feature pooling module in an action recognition model according to an embodiment of this application. Figure 5 As shown, for multiple input features of the multi-feature pooling module, they are concatenated (e.g., using cat concatenation), and then max pooling, mean pooling, and mode pooling operations are performed respectively. The features obtained from each pooling operation are then concatenated (e.g., using cat concatenation). Subsequently, a convolution module is used to adjust the channels, and pointwise multiplication and pointwise addition are used to aggregate the multiple features into a single feature vector. Finally, max pooling is used to optimize the feature vector, thus obtaining the output features of the multi-feature pooling module.
[0084] In the embodiments of this application, the multi-feature pooling block aggregates the features extracted by the convolutional neural network into a feature vector. The idea comes from human visual perception of actions. Considering that the sequential relationship of actions is easy to identify, there is no need to deliberately model the temporal relationship. The temporal relationship can be extracted and utilized by the optimization of the deep neural network itself. The aggregated feature vector naturally contains the temporal relationship, and thus can be used for action recognition.
[0085] In the embodiments of this application, real-time processing of surveillance video data using an action recognition model can yield recognition results of personnel actions in the video. This allows for anomaly detection based on pre-set operations. When abnormal actions are detected, such as entering a sensitive area without authentication or exhibiting abnormal behavior during a sensitive operation (e.g., obstruction during authentication), an anomaly warning is triggered. This can be achieved through audible and visual alarms, pop-up window alarms, or other methods to report the anomaly to management personnel. Simultaneously, relevant operation information can be automatically recorded, including the time, location, personnel, and specific actions of the detected abnormal actions. Furthermore, reinforcement learning can be introduced to optimize and train the model. For example, for identified anomalies, the accuracy of the identification is manually assessed, labels are added, and the data is saved to a database. Then, during non-business hours, such as at night, the model learns from misidentified data to improve its accuracy.
[0086] Based on the above-described anomaly detection method, this application also provides an anomaly detection device. The following will be combined with... Figure 7 The device is described in detail.
[0087] Figure 7 A schematic block diagram of an anomaly detection device according to an embodiment of this application is shown.
[0088] like Figure 7 As shown, the anomaly detection device 700 of this embodiment includes a data acquisition module 710, an identity recognition module 720, a calculation module 730, an image extraction module 740, an action recognition module 750, and a detection module 760.
[0089] The data acquisition module 710 is used to acquire surveillance video data of a designated area. In one embodiment, the data acquisition module 710 can be used to perform the operation S210 described above, which will not be repeated here.
[0090] The identity recognition module 720 is used to identify the identity features of a person in a first target frame image in the surveillance video data. In one embodiment, the identity recognition module 720 can be used to perform the operation S220 described above, which will not be repeated here.
[0091] The calculation module 730 is used to determine the importance weight based on the identity characteristics and the regional characteristics of the set area, and to calculate the frame extraction ratio of the surveillance video data based on the product of the preset ratio parameter and the importance weight. In one embodiment, the calculation module 730 can be used to perform the operation S230 described above, which will not be repeated here.
[0092] The image extraction module 740 is used to extract N second target frame images from the surveillance video data based on the frame extraction ratio; where N is an integer greater than 1. In one embodiment, the image extraction module 740 can be used to perform the operation S240 described above, which will not be repeated here.
[0093] The action recognition module 750 is used to extract feature data from N second target frame images and recognize the feature data to obtain the action recognition result of the person. In one embodiment, the action recognition module 750 can be used to perform the operation S250 described above, which will not be repeated here.
[0094] The detection module 760 is used to determine the anomaly detection result based on the matching relationship between the set area, identity features, and action recognition results. In one embodiment, the detection module 760 can be used to perform the operation S260 described above, which will not be repeated here.
[0095] According to an embodiment of this application, the identity features include the person's identity recognition result and the person's clothing recognition result. The calculation module 730 is further configured to determine a first weight based on the identity recognition result; determine a second weight based on the matching relationship between the identity recognition result and the area feature permissions; determine a third weight based on the matching relationship between the identity recognition result and the clothing recognition result; and determine an importance weight based on the first weight, the second weight, the third weight, and the area weight corresponding to the area feature.
[0096] According to an embodiment of this application, the feature extraction module 720 is further configured to input the first target frame image into a shallow feature extraction module to obtain a first feature; input the first feature into M concatenated basic feature extraction modules to obtain a second feature; wherein M is an integer greater than 1; superimpose the first feature and the second feature to obtain a third feature; input the third feature into an identity recognition module to obtain an identity feature; wherein the basic feature extraction module includes multiple parallel feature extraction channels, and the feature extraction channel includes a downsampling module, a depth convolution module and an upsampling module connected in sequence, wherein the downsampling module and the upsampling module in each feature extraction channel have the same sampling factor, and the downsampling module and the upsampling module between each feature extraction channel have different sampling factors.
[0097] According to an embodiment of this application, the basic feature extraction module further includes a convolutional channel mixing module, which includes a first convolutional module, a nonlinear mapping module, and a second convolutional module connected in sequence, wherein the convolutional kernel sizes of the first convolutional module and the second convolutional module are different.
[0098] According to an embodiment of this application, the action recognition module 750 is further configured to perform multiple attention operations sequentially on each second target frame image to obtain multi-level attention features of the second target frame image and intermediate attention features of each attention operation; fuse the multi-level attention features corresponding to each second target frame image to obtain a first fused feature; fuse the intermediate attention features corresponding to each second target frame image and the first fused feature to obtain a second fused feature; input the first fused feature into a multi-scale feature fusion module to obtain a first multi-scale fused feature; the multi-scale feature fusion module is configured to acquire local features and global features of the input features; input the second fused feature into the multi-scale feature fusion module to obtain a second multi-scale fused feature; fuse the first multi-scale fused feature and the second multi-scale fused feature to obtain a third multi-scale fused feature; input the third multi-scale fused feature into the action recognition module to obtain an action recognition result.
[0099] According to an embodiment of this application, the multi-scale feature fusion module includes a segmentation module, a pooling module, and a fusion module. The segmentation module is used to segment the input features of the multi-scale feature fusion module according to different ratios to obtain multiple segmented features. The pooling module is used to perform global average pooling and global max pooling operations on each segmented feature respectively. The fusion module is used to superimpose the output features of the pooling module to generate the output features of the multi-scale feature fusion module.
[0100] According to an embodiment of this application, the action recognition module 750 is further configured to concatenate multi-level attention features to obtain a first concatenated feature; perform max pooling, mean pooling, and mode pooling operations on the concatenated features respectively to obtain corresponding max pooling features, mean pooling features, and mode pooling features; concatenate the max pooling features, mean pooling features, and mode pooling features to obtain a second concatenated feature; and aggregate the second concatenated feature to obtain a first fused feature.
[0101] According to embodiments of this application, any multiple modules among the data acquisition module 710, identity recognition module 720, calculation module 730, image extraction module 740, action recognition module 750, and detection module 760 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data acquisition module 710, identity recognition module 720, calculation module 730, image extraction module 740, action recognition module 750, and detection module 760 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the data acquisition module 710, identity recognition module 720, calculation module 730, image extraction module 740, action recognition module 750, and detection module 760 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.
[0102] Figure 8 A block diagram of an electronic device according to an embodiment of the present application for an anomaly detection method is shown schematically.
[0103] like Figure 8 As shown, an electronic device 800 according to an embodiment of this application includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.
[0104] RAM 803 stores various programs and data required for the operation of electronic device 800. Processor 801, ROM 802, and RAM 803 are interconnected via bus 804. Processor 801 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 802 and / or RAM 803. It should be noted that the programs may also be stored in one or more memories other than ROM 802 and RAM 803. Processor 801 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.
[0105] According to embodiments of this application, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to a bus 804. The electronic device 800 may also include one or more of the following components connected to the input / output (I / O) interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.
[0106] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.
[0107] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 802 and / or RAM 803 and / or one or more memories other than ROM 802 and RAM 803 described above.
[0108] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the anomaly detection method provided in the embodiments of this application.
[0109] When the computer program is executed by the processor 801, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0110] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 809, and / or installed from a removable medium 811. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0111] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.
[0112] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0114] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.
Claims
1. An anomaly detection method characterized by, include: Acquire surveillance video data for a designated area; For the first target frame image in the surveillance video data, identify the identity features of the person in the first target frame image; The importance weight is determined based on the identity features and the regional features of the set area, and the frame extraction ratio of the surveillance video data is calculated based on the product of the preset ratio parameter and the importance weight. Based on the frame extraction ratio, N second target frame images are extracted from the surveillance video data; where N is an integer greater than 1. Extract feature data from N second target frame images, and identify the action recognition result of the person by recognizing the feature data; Anomaly detection results are determined based on the matching relationship between the defined region, the identity features, and the action recognition results.
2. The method of claim 1, wherein, The identity features include the person's identity recognition result and the person's clothing recognition result. The step of determining the importance weight based on the identity features and the regional features of the defined area includes: The first weight is determined based on the identity recognition result; The second weight is determined based on the matching relationship between the identity recognition result and the regional features; A third weight is determined based on the matching relationship between the identity recognition result and the clothing recognition result; The importance weight is determined based on the first weight, the second weight, the third weight, and the regional weight corresponding to the regional feature.
3. The method of claim 1, wherein, The identification of the identity features of the person in the first target frame image includes: The first target frame image is input into the shallow feature extraction module to obtain the first feature; The first feature is input into M concatenated basic feature extraction modules to obtain the second feature; where M is an integer greater than 1. The first feature and the second feature are superimposed to obtain the third feature; The third feature is input into the identity recognition module to obtain the identity feature; The basic feature extraction module includes multiple parallel feature extraction channels. Each feature extraction channel includes a downsampling module, a depthwise convolution module, and an upsampling module connected in sequence. The downsampling module and the upsampling module in each feature extraction channel have the same sampling factor, while the sampling factors of the downsampling module and the upsampling module are different between different feature extraction channels.
4. The method of claim 3, wherein, The basic feature extraction module also includes a convolutional channel mixing module, which comprises a first convolutional module, a nonlinear mapping module, and a second convolutional module connected in sequence. The first convolutional module and the second convolutional module have different kernel sizes.
5. The method of claim 1, wherein, The step of extracting feature data from N second target frame images and identifying the feature data to obtain the action recognition result of the person includes: For each second target frame image, perform multiple attention operations sequentially to obtain the multi-level attention features of the second target frame image and the intermediate attention features of each attention operation; The multi-level attention features corresponding to each of the second target frame images are fused to obtain the first fused feature; The intermediate attention features corresponding to each of the second target frame images are fused with the first fused feature to obtain the second fused feature; The first fusion feature is input into a multi-scale feature fusion module to obtain a first multi-scale fusion feature; the multi-scale feature fusion module is configured to obtain local features and global features of an input feature; The second fusion feature is input into the multi-scale feature fusion module to obtain a second multi-scale fusion feature; The first multi-scale fusion feature and the second multi-scale fusion feature are fused to obtain a third multi-scale fusion feature; The third multi-scale fusion feature is input into a motion recognition module to obtain a motion recognition result.
6. The method of claim 5, wherein, The multi-scale feature fusion module comprises a segmentation module, a pooling module and a fusion module; the segmentation module is configured to segment an input feature of the multi-scale feature fusion module according to different proportions to obtain a plurality of segmented features; the pooling module is configured to perform global average pooling and global maximum pooling operations on each of the segmented features respectively; and the fusion module is configured to superimpose output features of the pooling module to generate output features of the multi-scale feature fusion module.
7. The method of claim 5, wherein, The first fusion feature is obtained by fusing the multi-level attention features corresponding to the second target frame images, comprising: The first spliced feature is obtained by splicing the multi-level attention features; The maximum pooling feature, the mean pooling feature and the mode pooling feature corresponding to the spliced feature are obtained by performing maximum pooling, mean pooling and mode pooling operations on the spliced feature respectively; The second spliced feature is obtained by splicing the maximum pooling feature, the mean pooling feature and the mode pooling feature; The first fusion feature is obtained by aggregating the second spliced feature.
8. An abnormality detection device characterized by comprising: Comprise: A data acquisition module configured to acquire monitoring video data of a set region; An identity recognition module configured to recognize an identity feature of a person in a first target frame image in the monitoring video data; A calculation module configured to determine an importance weight according to the identity feature and a region feature of the set region, and calculate a frame extraction ratio for the monitoring video data according to a product of a preset proportion parameter and the importance weight; An image extraction module configured to extract N second target frame images from the monitoring video data based on the frame extraction ratio; wherein N is an integer greater than 1; A motion recognition module configured to extract feature data of the N second target frame images, and recognize the feature data to obtain a motion recognition result of the person; A detection module configured to determine an abnormality detection result according to a matching relationship among the set region, the identity feature and the motion recognition result.
9. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1-7.
10. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-7.
11. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the steps of the method according to any one of claims 1-7.