Operation record generation method and device, equipment and storage medium

Abstract

The invention provides an operation record generation method and device, equipment and a storage medium, and relates to the technical field of biometric authentication. The method comprises the steps that a detection result output by a living body detection model is acquired, and the living body detection model is obtained by training a plurality of acquired sample figure images and annotation information of each sample figure image. And then, if the detection result indicates that the living body exists in the operation area, performing feature matching on the first face feature of each image in the plurality of continuous images and a preset face feature library. And determining whether target face features matched with the continuous images exist in a preset face feature library or not, and if yes, generating a first operation record of the electronic equipment. According to the method, the accuracy and efficiency of troubleshooting of the electronic equipment are improved, and the safety requirement of the electronic equipment is met to a certain extent.

Description

Technical Field

[0001] The present application relates to the field of biometric authentication technology, and in particular to an operation record generation method, device, equipment and storage medium. Background Art

[0002] Operation and maintenance services and stress testing are important means to ensure the normal operation of electronic equipment, improve performance, ensure safety and extend life. However, in the actual operation and maintenance and testing process, some faults may occur that are difficult to locate or diagnose.

[0003] When problems occur during operation and testing, the existing problem location method is to query the log through alarm information, and then analyze the operation records of the electronic equipment through log analysis. However, in some scenarios, there is no corresponding operation record in the log. At this time, it is necessary to call the monitoring to confirm whether there is human operation that causes the operation and testing failure.

[0004] However, the prior art has technical problems of low accuracy and efficiency in troubleshooting electronic equipment. Summary of the invention

[0005] The present application provides an operation record generation method, apparatus, device and storage medium to solve the problems of low accuracy and efficiency in troubleshooting of electronic equipment in the prior art.

[0006] In a first aspect, the present application provides an operation record generation method, including:

[0007] Acquire a detection result output by a living body detection model, wherein the detection result is used to indicate whether a living body exists in an operation area of ​​the electronic device;

[0008] If the detection result indicates that there is a living body in the operation area, feature matching is performed on the first facial feature of each image in the multiple continuous images of the operation area with a preset facial feature library to determine whether there is a target facial feature matching the multiple continuous images in the preset facial feature library; the preset facial feature library is used to store the identification of authorized personnel who can operate the electronic device and the second facial feature of each authorized person;

[0009] If it is determined that there is a target facial feature matching the multiple continuous images in the preset facial feature library, a first operation record of the electronic device is generated, and the first operation record includes an identifier of a target authorized person corresponding to the target facial feature.

[0010] In a possible design, obtaining the detection result output by the liveness detection model includes:

[0011] Acquiring a plurality of continuous images of the operation area;

[0012] The multiple continuous images are input into a living body detection model to obtain a detection result output by the living body detection model.

[0013] In a possible design, acquiring a plurality of continuous images of the operating area includes:

[0014] When the light-sensing camera senses that the light change degree in the operation area exceeds a preset light change degree, the plurality of continuous images are collected by the light-sensing camera.

[0015] In a possible design, obtaining the detection result output by the liveness detection model includes:

[0016] Generate a confidence score for each image, where the confidence score is used to indicate whether the image contains a living body;

[0017] The detection result is output according to the confidence of each image and the preset confidence.

[0018] In one possible design, the method further includes:

[0019] If it is determined that the target facial features matching the plurality of consecutive images do not exist in the preset facial feature library, controlling the electronic device to display a login interface;

[0020] Acquire user account information input by the user into the electronic device through the login interface;

[0021] Authentication is performed on the user account information to generate a verification result;

[0022] If the verification result indicates that the user account information passes identity authentication, a second operation record of the electronic device is generated, and the second operation record includes the account number in the user account information.

[0023] In one possible design, the method further includes:

[0024] If the verification result indicates that the user account information has not passed identity authentication, a third operation record of the electronic device is generated, and the third operation record includes any one of the multiple continuous images.

[0025] In a possible design, before obtaining the detection result output by the liveness detection model, the method further includes:

[0026] Acquire multiple sample person images and annotation information of each sample person image, wherein the annotation information is used to indicate whether the person included in the sample person image is a living person or a prosthesis;

[0027] According to the multiple sample person images and the annotation information of each sample person image, the initial model is trained to obtain a liveness detection model; the initial model includes a fast Fourier transform module, a real part processing module, an imaginary part processing module and a feature fusion module, the real part processing module includes a first self-attention unit and a first bidirectional long short-term memory network unit, and the imaginary part processing module includes a second self-attention unit and a second bidirectional long short-term memory network unit;

[0028] Among them, the output end of the fast Fourier transform module is respectively connected to the input end of the real part processing module, the input end of the imaginary part processing module and the input end of the feature fusion module, and the output end of the real part processing module and the output end of the imaginary part processing module are respectively connected to the input end of the feature fusion module.

[0029] In a second aspect, the present application provides an operation record generating device, comprising:

[0030] An acquisition module, used to acquire a detection result output by a living body detection model, wherein the detection result is used to indicate whether there is a living body in an operation area of ​​the electronic device;

[0031] a matching module, configured to perform feature matching between a first face feature of each of a plurality of consecutive images of the operation area and a preset face feature library if the detection result indicates that a living body exists in the operation area;

[0032] A determination module, used to determine whether there is a target facial feature matching the plurality of continuous images in the preset facial feature library; the preset facial feature library is used to store the identification of authorized personnel who can operate the electronic device and the second facial feature of each authorized personnel;

[0033] A generation module is used to generate a first operation record of the electronic device if it is determined that there is a target facial feature matching the multiple continuous images in the preset facial feature library, and the first operation record includes an identification of a target authorized person corresponding to the target facial feature.

[0034] In a possible design, the acquisition module is further used to:

[0035] Acquiring a plurality of continuous images of the operation area;

[0036] The multiple continuous images are input into a living body detection model to obtain a detection result output by the living body detection model.

[0037] In a possible design, the acquisition module also includes: a collection module, which is used to collect the multiple continuous images through the light-sensing camera when the light-sensing camera senses that the light change degree of the operating area exceeds a preset light change degree.

[0038] In a possible design, the generation module is also used to generate a confidence level for each image, where the confidence level is used to indicate whether the image contains a living body; and the detection result is output based on the confidence level of each image and a preset confidence level.

[0039] In a possible design, the operation record generating device further includes: a control module, configured to control the electronic device to display a login interface if it is determined that there is no target facial feature matching the plurality of continuous images in the preset facial feature library;

[0040] The acquisition module is further used to acquire user account information input into the electronic device by the user through the login interface;

[0041] The generation module is also used to authenticate the user account information and generate a verification result; if the verification result indicates that the user account information passes the identity authentication, a second operation record of the electronic device is generated, and the second operation record includes the account number in the user account information.

[0042] In one possible design, the generation module is further used to generate a third operation record of the electronic device if the verification result indicates that the user account information has not passed the identity authentication, and the third operation record includes any one of the multiple continuous images.

[0043] In a possible design, the operation record generating device further includes: a training module,

[0044] The acquisition module is further used to acquire a plurality of sample person images and annotation information of each sample person image, wherein the annotation information is used to indicate whether the person included in the sample person image is a living person or a prosthesis;

[0045] The training module is used to perform model training on the initial model according to the multiple sample person images and the annotation information of each sample person image to obtain a liveness detection model; the initial model includes a fast Fourier transform module, a real part processing module, an imaginary part processing module and a feature fusion module, the real part processing module includes a first self-attention unit and a first bidirectional long short-term memory network unit, and the imaginary part processing module includes a second self-attention unit and a second bidirectional long short-term memory network unit;

[0046] Among them, the output end of the fast Fourier transform module is respectively connected to the input end of the real part processing module, the input end of the imaginary part processing module and the input end of the feature fusion module, and the output end of the real part processing module and the output end of the imaginary part processing module are respectively connected to the input end of the feature fusion module.

[0047] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method described in the first aspect above and various possible designs.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs is implemented.

[0049] The present application provides an operation record generation method, device, equipment and storage medium, which obtains the detection result output by the liveness detection model. If the detection result indicates that there is a live body in the operation area, the detected live body is authenticated. The specific authentication process is to match the first facial feature of each image in multiple continuous images with a preset facial feature library, and then determine whether there is a target facial feature matching the multiple continuous images in the preset facial feature library. If so, the first operation record of the electronic device is generated. During the entire liveness detection and authentication process, regardless of whether a live body is detected or not, whether the authentication is passed, the system will generate a corresponding operation record log to prevent the failure of the electronic device from being difficult to determine whether there is interference from human operation. When a failure occurs in the electronic device, the relevant operation record log can be queried in the first time to accurately locate the failure problem, thereby improving the accuracy and efficiency of troubleshooting of the electronic device. In addition, only authenticated live bodies can operate the electronic device, thereby strengthening the security protection of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 Schematic diagram of the process of generating operation records provided in the embodiment of the present application Figure 1 ;

[0052] Figure 2 A schematic diagram of interaction of an electronic device provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of the structure of a liveness detection model provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of the system operation of the operation record generation method provided in an embodiment of the present application;

[0055] Figure 5 Schematic diagram of the process of generating operation records provided in the embodiment of the present application Figure 2 ;

[0056] Figure 6 A schematic diagram of the structure of the operation record generating device provided by this application;

[0057] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0058] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0060] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0061] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0062] Electronic devices are widely used to solve various computing tasks and provide a variety of services. For some electronic devices that require centralized management, efficient heat dissipation and security protection, these electronic devices and other network devices (such as switches and routers) can be installed in the cabinets of data centers or computer rooms in an orderly and standardized manner to maximize the space utilization of data centers or computer rooms.

[0063] The installation process includes physical installation, connecting power and network cables, etc. Once the electronic equipment is installed, operation and maintenance services and stress testing are required. Operation and maintenance services include daily monitoring, maintenance and management to ensure the normal operation of electronic equipment. Stress testing is to verify the performance and stability of electronic equipment under high load conditions to ensure that it can withstand the expected workload in actual use.

[0064] However, in the actual operation and testing process, some faults that are difficult to locate or diagnose may occur. These faults may be caused by hardware failures, software errors, network problems or other complex reasons.

[0065] Developers need to locate faults that occur during the operation and testing of electronic equipment to ensure the normal operation of the electronic equipment. In the prior art, when problems occur during the operation and testing of electronic equipment, the relevant logs are first queried through the alarm system. These logs record various events and error information generated during the operation of the electronic equipment.

[0066] By analyzing the logs, we can understand the operation records of electronic devices, which usually include the system's automatic operations, user operations, and software running conditions.

[0067] However, in some specific scenarios, the log may not record certain operations on the electronic device. For example, if the operator does not have sufficient authority to record the log, some operations may not be recorded. Or, an abnormal user may deliberately delete or tamper with the log to hide the traces of his operation, etc.

[0068] When key operation records are missing from the log, the monitoring of the data center or computer room is retrieved to confirm whether there is human operation that causes the operation and testing failure. Since monitoring has blind spots, it is impossible to conduct comprehensive and accurate monitoring of a single electronic device, and the monitoring data obtained may not be complete, which affects the location and comprehensive analysis of the fault problem.

[0069] In addition, certain human errors in data centers or computer rooms (such as plugging and unplugging hard drives, network cables, pressing buttons, etc.) are difficult to completely avoid and may cause electronic equipment to be exposed to unexpected attacks, such as data leakage or system attacks.

[0070] It can be seen that due to the lack of operation records in the logs of the existing technology in some cases, it is difficult to track and locate the root cause of the fault problem, which not only affects the accuracy of fault location, but also prolongs the time of troubleshooting, resulting in low troubleshooting efficiency. In addition, human error and abnormal operation can also threaten the safety of electronic equipment.

[0071] In response to the above technical problems, the inventors found that the operation records in the log are crucial to the troubleshooting process of electronic equipment, and the monitoring of the data center or computer room will also make the operation records in the log incomplete due to the existence of blind spots. Based on this, the inventors designed a light-sensing liveness recognition system, which performs liveness recognition through liveness detection sensors and related face recognition algorithms. When a living body is identified, the system log is uploaded by recording information and capturing photos to prevent the failure of the electronic equipment from being difficult to determine whether there is interference from human operation. In addition, in order to prevent human misoperation and abnormal operation, a protection module is installed for the electronic equipment, and a face recognition system is added to unlock the protection module to strengthen the security protection of the electronic equipment.

[0072] It should be explained that the optical liveness recognition system includes an optical camera deployed on the protection module, and a system that interacts with the optical camera. Among them, the system that interacts with the optical camera includes a driver, an operating program, and a central processing unit (CPU). The system that interacts with the optical camera in the embodiment of the present application includes but is not limited to a computer system, which is not specifically limited.

[0073] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0074] An embodiment of the present application provides a method for generating an operation record. Figure 1 Schematic diagram of the process of generating operation records provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the operation record generation method includes:

[0075] S101: Obtain the detection result output by the living body detection model.

[0076] Among them, the detection result is used to indicate whether there is a living body in the operating area of ​​the electronic device. It can be understood that the liveness detection model is used to distinguish between a live body and a fake body. It plays a vital role in biometric systems, especially in applications such as facial recognition, fingerprint recognition, and iris recognition.

[0077] This embodiment uses the light-sensing camera, Fresnel filter, and infrared sensor to work together to input multiple continuous images into the liveness detection model, thereby obtaining corresponding detection results. Specifically, the specific steps for obtaining the detection results output by the liveness detection model are as follows:

[0078] Step a1: Acquire multiple continuous images of the operation area.

[0079] First of all, it is necessary to briefly explain the hardware structure and interaction relationship of the light-sensing camera, infrared sensor, Fresnel filter, and protection module of the electronic device.

[0080] The light-sensing camera is deployed on the protection module of the electronic device and can detect changes in light in the environment and capture images.

[0081] Infrared sensors are used to detect infrared radiation and can detect heat sources (such as humans or animals) in low-light or no-light environments. Infrared sensors and light-sensing cameras can be designed as a module and integrated into the same housing to detect both visible light and infrared light. Infrared sensors can also be installed next to light-sensing cameras or independently.

[0082] Fresnel filter is a specially designed optical element used to focus or enhance light of a specific wavelength, helping light-sensing cameras or infrared sensors to more effectively perceive light changes or light wavelengths. It is usually installed in front of light-sensing cameras or infrared sensors to improve light detection capabilities.

[0083] Furthermore, Figure 2 The interactive schematic diagram of the electronic device provided in the embodiment of the present application is as follows: Figure 2 As shown in the figure, the system uses the Universal Serial Bus (USB) interface to connect the light-sensing camera, infrared sensor, and protection module of the electronic device to the CPU to realize data transmission and processing. This architecture is widely used in the fields of security monitoring, automatic control, and environmental monitoring. It realizes intelligent monitoring and management by real-time monitoring and responding to environmental changes. Among them, the system supports both on-board light-sensing cameras directly integrated on the protection module and external light-sensing cameras connected to the protection module through the USB interface.

[0084] In one possible implementation, the light-sensing camera uses a Fresnel filter to enhance its ability to detect light and continuously monitors the ambient light within its field of view (i.e., the operating area of ​​the electronic device). The processing unit of the light-sensing camera analyzes the collected light data in real time, and starts capturing multiple continuous images when the degree of light change exceeds a preset degree of light change.

[0085] In another possible implementation, the infrared sensor continuously monitors the infrared light in its field of view (i.e., the operating area of ​​the electronic device) by focusing or enhancing infrared light through a Fresnel filter. When the infrared sensor detects infrared light of a specific wavelength (10 μm), an electrical signal is generated. The infrared sensor transmits the electrical signal to the system, and after the system receives the electrical signal, it triggers the light-sensing camera to start capturing multiple continuous images.

[0086] Explanatory, infrared light with a wavelength of 10μm is generally in the far infrared range and is often used for thermal imaging and temperature detection. Therefore, the electrical signal transmitted by the infrared sensor indicates the presence of a heat source (such as a human body or animal) within the field of view.

[0087] It should be understood that the system can wake up the light-sensing camera to capture images when the light-sensing camera detects a change in light or the infrared sensor detects an infrared signal, and put the light-sensing camera into sleep mode when no light change or infrared signal is detected, thereby significantly reducing the power consumption of the entire machine.

[0088] Step a2: input multiple continuous images into the liveness detection model to obtain the detection results output by the liveness detection model.

[0089] Specifically, the liveness detection model in this step is obtained through training, and the model training process is as follows:

[0090] Step b1: obtaining a plurality of sample person images and annotation information of each sample person image.

[0091] The annotation information is used to indicate whether the person contained in the sample person image is a living person (i.e., a real person) or a prosthesis (such as a portrait in a photo or video, a model, etc.).

[0092] Step b2: training the initial model based on multiple sample person images and the annotation information of each sample person image to obtain a liveness detection model.

[0093] The initial model is an untrained or preliminarily trained model, usually a neural network or other machine learning model. The training of the initial model includes inputting sample person images and their corresponding annotation information, and adjusting the parameters of the initial model so that it can more accurately predict whether the person in the actual image is alive or fake.

[0094] In a possible implementation, the liveness detection model of this embodiment adopts an improved bidirectional long short-term memory network model, and the corresponding initial model adopts a bidirectional long short-term memory network model.

[0095] It should be explained that the initial model includes a fast Fourier transform module, a real part processing module, an imaginary part processing module and a feature fusion module. The real part processing module includes a first bidirectional long short-term memory network unit and a first self-attention unit, and the imaginary part processing module includes a second bidirectional long short-term memory network unit and a second self-attention unit.

[0096] Further, the output end of the fast Fourier transform module is respectively connected to the input end of the real part processing module, the input end of the imaginary part processing module and the input end of the feature fusion module. The output end of the real part processing module and the output end of the imaginary part processing module are respectively connected to the input end of the feature fusion module.

[0097] By training the bidirectional long short-term memory network model, an improved bidirectional long short-term memory network model capable of liveness detection is obtained. The improved bidirectional long short-term memory network model can not only handle the liveness detection problem by switching from the spatial domain to the frequency domain, but also use two bidirectional long short-term memory network processing methods for residual connection. In addition, the model can be forced to converge faster by mixing different loss functions in two channels.

[0098] Next, the improved bidirectional long short-term memory network model will be explained in detail. Figure 3 A schematic diagram of the structure of the liveness detection model provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, first, multiple continuous images are converted from the spatial domain to the frequency domain through the fast Fourier transform module for calculation, and the real part calculation results and the imaginary part calculation results are obtained. Among them, the real part calculation result is related to the symmetry and amplitude information of the image; the imaginary part calculation result is related to the phase information of the image.

[0099] In order to make full use of the features of the real part calculation results and the imaginary part calculation results, the obtained real part calculation results and the imaginary part calculation results are respectively transmitted to the corresponding real part processing module and the imaginary part processing module in the form of dual channels. Among them, the first bidirectional long short-term memory network unit of the real part processing module and the second bidirectional long short-term memory network unit of the imaginary part processing module can respectively capture the time dependency in the real part calculation result sequence and the imaginary part calculation result sequence.

[0100] In both the real part processing module and the imaginary part processing module, two bidirectional long short-term memory network processing methods are used to process the corresponding real part calculation results and imaginary part calculation results to obtain processed image features.

[0101] Taking the real part processing module as an example, the above processing process is explained. After the real part processing module receives the real part calculation result transmitted by the fast Fourier transform module, the first processing method is to directly extract the first image feature of the real part calculation result through the first bidirectional long short-term memory network unit. The second processing method is to process the real part calculation result through the first bidirectional long short-term memory network unit to obtain a processed feature sequence, and the feature sequence is sequentially subjected to a convolution operation, a first self-attention unit, an activation function, and a pooling operation to obtain a processed second image feature.

[0102] The processing process of the imaginary part processing module is similar to that of the real part processing module, and will not be described in detail. The imaginary part calculation result is outputted as the corresponding third image feature and fourth image feature after passing through the imaginary part processing module.

[0103] It is worth noting that in order to effectively extract the features of different parts of the image, a multi-scale convolution operation can be used. Exemplarily, the real part processing module can use a dilated convolution operation, because the real part calculation result contains more global structures and large-scale features, and the dilated convolution operation can effectively capture this information without losing details. The imaginary part processing module can use a standard convolution operation, because the imaginary part calculation result contains more local changes and detail information, and the standard convolution operation is more suitable for capturing these details. In addition, since convolution kernels of different sizes can capture different details and patterns in the image, the convolution kernel sizes used during the convolution operation can include 1*1, 2*2, and 3*3.

[0104] After the convolution operation, the output corresponding to the convolution operation is passed to the first (second) self-attention unit of the real (imaginary) part processing module, which can significantly enhance the feature extraction ability and global information capture ability of the real (imaginary) part processing module.

[0105] Next, the output of the first (second) self-attention unit is passed to the "normalization + activation" unit, and the output of the "normalization + activation" unit is pooled to reduce the feature dimension of the image. It can be understood that the normalization operation improves numerical stability, while the activation function (such as the rectified linear unit and the Gaussian error linear unit) introduces nonlinearity, enabling the liveness detection model to better learn and represent complex patterns and relationships. This combination of normalization and activation function can significantly improve the performance and generalization ability of the liveness detection model.

[0106] Afterwards, the first image features and the third image features extracted by the first bidirectional long short-term memory network unit and the second bidirectional long short-term memory network unit, as well as the second image features and the fourth image features after the pooling operation, are input into the feature fusion module together.

[0107] Explanatory, feature fusion is to combine features from different sources or different levels to form a comprehensive feature representation. The feature fusion module in the liveness detection model of this embodiment uses the idea of ​​residual network to perform feature fusion on the first image feature, the second image feature, the third image feature, and the fourth image feature to obtain a comprehensive image feature. This solves the gradient vanishing and gradient exploding problems in the real part processing module and the imaginary part processing module to ensure the robustness of the liveness detection model.

[0108] It should be noted that in some cases, the liveness detection model may need to process classification tasks and regression tasks at the same time. By using different loss functions in different modules, the model can be effectively optimized. In one possible implementation, in order to make the liveness detection model converge more quickly, two loss functions are mixed in the model training process. Among them, the cross entropy loss function is used in the real part processing module, and the mean square loss function is used in the imaginary part processing module.

[0109] The loss function is often used in combination with the Sigmoid function to make the gradient calculation more concise and stable. Since the output range of the Sigmoid function is between [0,1], it can prevent the output value from being too large or too small, thereby avoiding the problem of unstable output, and can effectively measure the difference between the predicted probability and the actual label.

[0110] Therefore, the model training process also includes passing the comprehensive image features output by the feature fusion module into the Sigmoid function through the fully connected layer, and determining the detection result output by the liveness detection model according to the output of the Sigmoid function.

[0111] It can be understood that since the output range of the Sigmoid function is between [0,1], which is consistent with the value range of the confidence level, the output of the Sigmoid function can be interpreted as the probability or confidence level that the model predicts that the input sample belongs to a certain category.

[0112] For example, if the output of Sigmoid is close to 1, it means that the confidence of the liveness detection model is high that the image contains a live object. If the output of the Sigmoid function is close to 0, it means that the confidence of the model is high that the image contains a fake object. If the output of the Sigmoid function is close to 0.5, it means that the liveness detection model cannot determine whether the image contains a live object.

[0113] It can be seen that in a possible implementation, after multiple continuous images are input into the liveness detection model, the liveness detection model will generate a confidence level corresponding to each image to indicate whether the image contains a live body. Then, all the confidence levels obtained are comprehensively analyzed to calculate a comprehensive confidence level. Based on the comprehensive confidence level and the preset confidence level, a detection result is generated to indicate whether there is a live body in the operation area.

[0114] It should be understood that in poor light conditions, the light-sensing camera can obtain less image feature information. In this case, the model training can only use eye activities (such as blinking, eye movement, optical flow, etc.) for liveness detection to effectively reduce the amount of calculation and avoid the influence of irrelevant environment on the detection results. Through the convolution operation, it is possible to identify which areas in the image contain important features.

[0115] In one possible implementation, when the light is poor, eye activity is more important than activity in other areas. Therefore, during training, the weights of the convolution kernels are adjusted so that the features of the eye area occupy a larger proportion in the output.

[0116] It is worth noting that the detection results finally output by the liveness detection model include the presence of liveness and the absence of liveness. If there is no liveness, the collected image is directly saved to the corresponding local log and operating system log. If there is a liveness, the collected image is recorded and the info information is reported to the local log and operating system log. Among them, the info information records the normal operating status of the system, including but not limited to the detection time, detection results, operating environment, etc.

[0117] S102. If the detection result indicates that there is a living body in the operation area, feature matching is performed on the first facial feature of each image in the multiple consecutive images of the operation area with a preset facial feature library to determine whether there is a target facial feature in the preset facial feature library that matches the multiple consecutive images.

[0118] The preset facial feature library is used to store the identification of authorized persons who can operate electronic devices and the second facial feature of each authorized person.

[0119] It is understandable that if the detection result indicates that there is a living body in the operation area, the detected living body needs to be authenticated to confirm whether the detected living body has the authority to operate the electronic device. In a possible implementation, this embodiment uses a face recognition network (FaceNet) to extract facial features from each image in multiple consecutive images to obtain the first facial feature of each image.

[0120] Explanation: The face recognition network is a deep learning model for face recognition and verification. It maps each image into a high-dimensional feature vector space through a neural network. In this high-dimensional feature vector, the face recognition network converts the face area in the image into a feature vector (i.e., the first face feature), which contains various feature information that can represent the face.

[0121] In a possible implementation, after obtaining the first facial feature corresponding to each image, the first facial feature is input into a support vector machine (SVM) model. The SVM model outputs a judgment result based on the input first facial feature, indicating whether the living body corresponding to the first facial feature has the permission.

[0122] Specifically, the SVM model is a supervised learning model, which is commonly used for classification and regression tasks. It divides data points of different categories by finding an optimal hyperplane. In order to ensure that the SVM model can better handle facial features, the random forest strategy is used to train the SVM model. In a possible implementation, an integrated learner (such as VotingClassifier) ​​can be used to achieve the fusion of the random forest strategy and the SVM model.

[0123] Explanation: Random forest strategy is an ensemble learning method based on decision trees, which makes the final prediction by building multiple decision trees and voting. Random forest strategy can handle high-dimensional data and has good generalization ability, which can help extract and select important features, thereby improving the performance of SVM model. Specifically, random forest strategy can be used to preprocess the first face features, and the preprocessed first face features can be provided as input to the SVM model.

[0124] Next, the SVM model uses the knowledge learned during the training process to convert the preprocessed first face features into geometric features. These geometric features may include key points (such as the location of eyes, nose, and mouth), facial contours, and textures.

[0125] The constructed geometric features are compared with the second facial features pre-entered in the preset facial feature library. It should be understood that the second facial features in the preset facial feature library are also stored in the form of geometric features. By comparing the similarity between the constructed geometric features and the second facial features in the preset facial feature library, the prediction result of the first facial features is obtained. The similarity is usually expressed as a numerical value, reflecting the degree of similarity between the two sets of features.

[0126] It can be understood that each image in the multiple continuous images corresponds to a prediction result, and multiple prediction results will be obtained in the end. A weight is assigned to each prediction result through the soft voting mechanism, and then all the prediction results are weighted averaged to obtain a final prediction result. Further, it is determined based on the final prediction result whether there is a target face feature in the preset face feature library that matches the multiple continuous images.

[0127] Exemplarily, the final prediction result after weighted averaging is a binary classification result (0 or 1). If the final prediction result is 0, it means that the target facial features that match the multiple continuous images do not exist in the preset facial feature library, that is, the living body corresponding to the multiple continuous images does not have the authority to operate the electronic device. If the final prediction result is 1, it means that the target facial features that match the multiple continuous images exist in the preset facial feature library, that is, the living body corresponding to the multiple continuous images has the authority to operate the electronic device.

[0128] S103: If it is determined that there is a target facial feature matching the multiple consecutive images in the preset facial feature library, a first operation record of the electronic device is generated.

[0129] In a possible implementation, if there is a target facial feature matching multiple consecutive images in the preset facial feature library, a first operation record of the electronic device is generated, wherein the first operation record includes an identifier of a target authorized person corresponding to the target facial feature.

[0130] If the target facial features matching the multiple continuous images do not exist in the preset facial feature library, a third operation record of the electronic device is generated, and the electronic device is controlled to display a login interface, that is, the account and password verification process is performed. The third operation record includes any one of the multiple continuous images, and the login interface of the electronic device is deployed on the protection module.

[0131] Furthermore, the system obtains the user account information entered into the electronic device by the user through the login interface, and authenticates the user account information to generate a verification result. If the verification result indicates that the user account information passes the identity authentication, the system automatically generates a second operation record of the electronic device and sends a USB signal to unlock the protection module. The second operation record includes the account number in the user account information. If the verification result indicates that the user account information does not pass the identity authentication, the user's operation record is also generated and an alarm is issued.

[0132] The present application provides an operation record generation method. When the light-sensing camera senses that the light change degree of the operation area exceeds the preset light change degree, or the infrared sensor senses infrared light of a specific wavelength, a plurality of continuous images are collected by the light-sensing camera. Next, the plurality of continuous images are input into the liveness detection model, and the detection result output by the liveness detection model for indicating whether there is a live body in the operation area is obtained. Afterwards, if the detection result indicates that there is a live body in the operation area, the detected live body is authenticated. The specific authentication process is to match the first face feature of each image in the plurality of continuous images with the preset face feature library, and then determine whether there is a target face feature matching the plurality of continuous images in the preset face feature library. If there is, a first operation record of the electronic device is generated. If not, the electronic device is controlled by the system to display a login interface, and the user account information input into the electronic device by the user through the login interface is authenticated to generate a verification result. If the verification result indicates that the user account information passes the identity authentication, a second operation record including the user account information is generated; if the verification result indicates that the user account information fails the identity authentication, a third operation record including any one of the plurality of continuous images is generated. It can be seen that during the entire liveness detection and authentication process, the system will record all operations of the system itself, the light-sensing camera, the electronic equipment, and the protection module. When an electronic device fails, the relevant operation record log can be queried immediately to accurately locate the fault problem, thereby improving the accuracy and efficiency of troubleshooting of the electronic equipment. In addition, a protection module is installed for the electronic equipment. Only authenticated live bodies can unlock the protection module to operate the electronic equipment, thus strengthening the security protection of the electronic equipment.

[0133] In summary, through a specific embodiment Figure 1 The operation record generation method shown is summarized.

[0134] Figure 4 A system working diagram of the operation record generation method provided in an embodiment of the present application is shown as follows: Figure 4 As shown in the figure, after the light-sensing camera deployed on the protection module senses the light change, it sends the light data to the driver. The driver is responsible for logging and starting the light-sensing camera. When the driver receives multiple continuous images sent by the light-sensing camera again, it passes the multiple continuous images to the artificial intelligence algorithm for authentication. Among them, the artificial intelligence algorithm includes the living body detection model, face recognition network, random forest strategy, SVM model, etc. After authentication, the driver unlocks the protection module and grants operation permissions.

[0135] In one possible implementation, Figure 5 Schematic diagram of the process of generating operation records provided in the embodiment of the present application Figure 2 ,like Figure 5 As shown, the operation record generation method includes the following steps:

[0136] S501. Perform light sensing or infrared sensing through a light-sensing camera or an infrared sensor.

[0137] S502: When light changes or infrared light of a specific wavelength is detected, a light-sensing camera is started to capture multiple continuous images.

[0138] S503: Perform liveness detection on multiple continuous images using a liveness detection model.

[0139] If no living body is detected, S504 is executed, any one of the plurality of continuous images is selected for recording, and the process returns to S501. If a living body is detected, S505 is executed.

[0140] S505: Record the detected living body.

[0141] S506: Perform face recognition on the multiple continuous images, and determine whether there is a target face feature matching the multiple continuous images in the preset face feature library.

[0142] If the target facial feature exists, S507 is executed to record the facial features of the plurality of continuous images, and the process jumps to S512. If the target facial feature does not exist, S508 is executed.

[0143] S508: The system issues an alarm and records a log.

[0144] S509: The system controls the electronic device to display a login interface, and performs identity authentication on the user account information input by the user through the login interface.

[0145] If the identity authentication fails, S510 is executed, the system generates an alarm and records a log, and returns to S501. If the identity authentication passes, S511 is executed.

[0146] S511. Record user account information.

[0147] S512, unlock the protection module.

[0148] It can be seen that no matter the detection result output by the liveness detection model is that there is a live person in the operation area, or no live person in the operation area; or whether there is a target face feature that matches multiple consecutive images in the preset face feature library, the system will record all operations of the system itself, the light-sensing camera, the electronic device, and the protection module. Therefore, when an electronic device fails, the relevant operation record log can be queried immediately to accurately locate the failure problem, which improves the accuracy and efficiency of troubleshooting of electronic equipment.

[0149] Figure 6 This is a schematic diagram of the structure of the operation record generation device provided by this application. Figure 6 As shown, the operation record generating device 600 includes: an acquisition module 601, a matching module 602, a determination module 603, and a generation module 604;

[0150] The acquisition module 601 is used to obtain the detection result output by the living body detection model, and the detection result is used to indicate whether there is a living body in the operation area of ​​the electronic device;

[0151] A matching module 602 is configured to perform feature matching between a first face feature of each of a plurality of consecutive images of the operation area and a preset face feature library if the detection result indicates that a living body exists in the operation area;

[0152] The determination module 603 is used to determine whether there is a target facial feature matching the multiple consecutive images in a preset facial feature library; the preset facial feature library is used to store the identification of authorized personnel who can operate the electronic device and the second facial feature of each authorized personnel;

[0153] The generation module 604 is used to generate a first operation record of the electronic device if it is determined that there is a target facial feature matching multiple consecutive images in the preset facial feature library, and the first operation record includes an identifier of a target authorized person corresponding to the target facial feature.

[0154] In a possible design, the acquisition module 601 is further used for:

[0155] Acquire multiple sequential images of the operating area;

[0156] Multiple continuous images are input into the liveness detection model to obtain the detection results output by the liveness detection model.

[0157] In a possible design, the acquisition module 601 also includes: a collection module 605, which is used to collect multiple continuous images through the light-sensing camera when the light-sensing camera senses that the light change degree in the operation area exceeds a preset light change degree.

[0158] In a possible design, the generation module 604 is further used to generate a confidence level for each image, where the confidence level is used to indicate whether the image contains a living body; and output a detection result based on the confidence level of each image and a preset confidence level.

[0159] In a possible design, the operation record generating device 600 further includes: a control module 606, configured to control the electronic device to display a login interface if it is determined that the target facial feature matching the plurality of consecutive images does not exist in the preset facial feature library;

[0160] The acquisition module 601 is also used to obtain the user account information entered into the electronic device by the user through the login interface;

[0161] The generation module 604 is also used to authenticate the user account information and generate a verification result; if the verification result indicates that the user account information passes the identity authentication, a second operation record of the electronic device is generated, and the second operation record includes the account number in the user account information.

[0162] In one possible design, the generation module 604 is further used to generate a third operation record of the electronic device if the verification result indicates that the user account information has not passed the identity authentication, and the third operation record includes any one of the multiple consecutive images.

[0163] In a possible design, the operation record generating device 600 further includes: a training module 607;

[0164] The acquisition module 601 is further used to acquire a plurality of sample person images and annotation information of each sample person image, where the annotation information is used to indicate whether the person included in the sample person image is a living person or a prosthesis;

[0165] A training module 607 is used to perform model training on the initial model according to multiple sample person images and the annotation information of each sample person image to obtain a liveness detection model; the initial model includes a fast Fourier transform module, a real part processing module, an imaginary part processing module and a feature fusion module, the real part processing module includes a first self-attention unit and a first bidirectional long short-term memory network unit, and the imaginary part processing module includes a second self-attention unit and a second bidirectional long short-term memory network unit;

[0166] Among them, the output end of the fast Fourier transform module is respectively connected to the input end of the real part processing module, the input end of the imaginary part processing module and the input end of the feature fusion module, and the output end of the real part processing module and the output end of the imaginary part processing module are respectively connected to the input end of the feature fusion module.

[0167] The operation record generation device provided in the embodiments of the present application can be used to execute the operation record generation method in any of the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0168] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0169] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device may include: a transceiver 71 , a processor 72 , and a memory 73 .

[0170] The processor 72 executes the computer execution instructions stored in the memory, so that the processor 72 executes the scheme in the above embodiment. The processor 72 can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0171] The memory 73 is connected to the processor 72 via a system bus and completes communication between them. The memory 73 is used to store computer program instructions.

[0172] The transceiver 71 may be used to communicate with other devices.

[0173] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0174] The electronic device provided in the embodiments of the present application can be used to execute the method provided in any of the above embodiments. The implementation principles and technical effects are similar and will not be repeated here.

[0175] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the method provided in any of the above embodiments.

[0176] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0177] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0178] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0179] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0180] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0181] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0182] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0183] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0184] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0185] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for generating an operation record, characterized in that: include: Acquire a detection result output by a living body detection model, wherein the detection result is used to indicate whether a living body exists in an operation area of ​​the electronic device; If the detection result indicates that there is a living body in the operation area, feature matching is performed on the first facial feature of each image in the multiple continuous images of the operation area with a preset facial feature library to determine whether there is a target facial feature matching the multiple continuous images in the preset facial feature library; the preset facial feature library is used to store the identification of authorized personnel who can operate the electronic device and the second facial feature of each authorized person; If it is determined that there is a target facial feature matching the multiple continuous images in the preset facial feature library, a first operation record of the electronic device is generated, and the first operation record includes an identifier of a target authorized person corresponding to the target facial feature.

2. The method according to claim 1, characterized in that The obtaining of the detection result output by the living body detection model includes: Acquiring a plurality of continuous images of the operation area; The multiple continuous images are input into a living body detection model to obtain a detection result output by the living body detection model.

3. The method according to claim 2, characterized in that The acquiring of a plurality of continuous images of the operation area comprises: When the light-sensing camera senses that the light change degree in the operation area exceeds a preset light change degree, the plurality of continuous images are collected by the light-sensing camera.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of the detection result output by the living body detection model includes: Generate a confidence score for each image, where the confidence score is used to indicate whether the image contains a living body; The detection result is output according to the confidence of each image and the preset confidence.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: If it is determined that the target facial features matching the plurality of consecutive images do not exist in the preset facial feature library, controlling the electronic device to display a login interface; Acquire user account information input by the user into the electronic device through the login interface; Authentication is performed on the user account information to generate a verification result; If the verification result indicates that the user account information passes identity authentication, a second operation record of the electronic device is generated, and the second operation record includes the account number in the user account information.

6. The method according to claim 5, characterized in that The method further comprises: If the verification result indicates that the user account information has not passed identity authentication, a third operation record of the electronic device is generated, and the third operation record includes any one of the multiple continuous images.

7. The method according to any one of claims 1 to 3, characterized in that: Before obtaining the detection result output by the liveness detection model, the method further includes: Acquire multiple sample person images and annotation information of each sample person image, wherein the annotation information is used to indicate whether the person included in the sample person image is a living person or a prosthesis; According to the multiple sample person images and the annotation information of each sample person image, the initial model is trained to obtain a liveness detection model; the initial model includes a fast Fourier transform module, a real part processing module, an imaginary part processing module and a feature fusion module, the real part processing module includes a first self-attention unit and a first bidirectional long short-term memory network unit, and the imaginary part processing module includes a second self-attention unit and a second bidirectional long short-term memory network unit; Among them, the output end of the fast Fourier transform module is respectively connected to the input end of the real part processing module, the input end of the imaginary part processing module and the input end of the feature fusion module, and the output end of the real part processing module and the output end of the imaginary part processing module are respectively connected to the input end of the feature fusion module.

8. An operation record generating device, characterized in that: include: An acquisition module, used to acquire a detection result output by a living body detection model, wherein the detection result is used to indicate whether there is a living body in an operation area of ​​the electronic device; a matching module, configured to perform feature matching between a first face feature of each of a plurality of consecutive images of the operation area and a preset face feature library if the detection result indicates that a living body exists in the operation area; A determination module, used to determine whether there is a target facial feature matching the plurality of continuous images in the preset facial feature library; The preset facial feature library is used to store the identification of authorized personnel who can operate the electronic device and the second facial feature of each authorized personnel; A generation module is used to generate a first operation record of the electronic device if it is determined that there is a target facial feature matching the multiple continuous images in the preset facial feature library, and the first operation record includes an identification of a target authorized person corresponding to the target facial feature.

9. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images