State recognition method, device and equipment of safety equipment and storage medium
By combining satellite positioning technology, multimodal information of security equipment is obtained for feature fusion, which solves the problem of restricted recognition effect caused by traditional methods relying on a single mode, and improves the accuracy of security equipment status recognition.
Patent Information
- Application Number
- CN202411464841.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-03-07
AI Technical Summary
Traditional security equipment status recognition methods only rely on image modes, resulting in limited recognition effects and poor accuracy.
By combining satellite positioning technology, multimodal information of security equipment, including device image and device description information, feature fusion is performed, and fusion feature vectors are extracted for state recognition.
The accuracy of security equipment status recognition is improved, and the comprehensiveness of feature acquisition is enhanced by integrating multimodal information.
Smart Images
Figure CN119579948A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of security device management, and in particular to a method, apparatus, device and storage medium for identifying the status of a security device. Background Art
[0002] Currently, deep learning-based object detection methods (such as Faster R-CNN, YOLO, SSD, etc.) have been widely used in real-time scenarios. These methods can detect and identify objects in real time in video streams and are used in fields such as autonomous driving and security monitoring.
[0003] However, traditional methods usually rely only on the image modality for recognition, which results in the recognition effect being limited to the single factor of the image, making the recognition effect based on the image poor. Summary of the invention
[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for identifying the state of a security device. By applying satellite positioning to the state identification of the security device, the multimodal information of the security device can be integrated to extract a fused feature vector, thereby improving the comprehensiveness of feature acquisition. State identification is performed based on the fused feature vector, thereby improving the accuracy of state identification of the security device. The technical solution is as follows.
[0005] In one aspect, a method for identifying a state of a security device is provided, the method comprising: Acquire a device image of a security device and device description information of the security device; Extracting image features from the device image to obtain an image feature vector of the device image; Performing feature fusion on the image feature vector and the text feature vector generated based on the device description information to obtain a fused feature vector; A state classification result of the safety device is determined based on the fused feature vector; the state classification result of the safety device is used to indicate whether the safety device is in a normal state or an abnormal state.
[0006] In another aspect, a state identification device for a safety device is provided, the device comprising: A first information acquisition module, used to acquire a device image of a security device and device description information of the security device; A feature extraction module, used to extract image features from the device image to obtain an image feature vector of the device image; A feature fusion module, used for performing feature fusion on the image feature vector and the text feature vector generated based on the device description information to obtain a fused feature vector; A classification module is used to determine a state classification result of the safety device based on the fused feature vector; the state classification result of the safety device is used to indicate whether the safety device is in a normal state or an abnormal state.
[0007] In a possible implementation manner, the device further includes: A second information acquisition module, used to acquire environmental information of the security device; A preprocessing module, used for performing image preprocessing on the device image based on the environmental information to obtain a preprocessed device image; The feature extraction module is used to extract image features from the preprocessed device image to obtain an image feature vector of the device image.
[0008] In a possible implementation, the feature extraction module is used to: Inputting the device image into a pre-trained image feature extraction model to obtain an image feature vector of the device image output by the image feature extraction model; Among them, the image feature extraction model is a model obtained after fine-tuning the model through the second training sample set based on the model pre-trained based on the first training sample set; the scale of the first training sample set is larger than the scale of the second training sample set; the second training sample set contains security device sample images and the first image labels corresponding to the security device sample images.
[0009] In a possible implementation, the feature fusion module is used to: Inputting the image feature vector and the device description information into a pre-trained feature fusion model to obtain fusion features output by the feature fusion model; Among them, the feature fusion model is a model obtained after fine-tuning the model through the fourth training sample set based on the model pre-trained based on the third training sample set; the scale of the third training sample set is larger than the scale of the fourth training sample set; the fourth training sample set contains image feature vectors and device description information of security device sample images, as well as the second image label corresponding to the security device sample image.
[0010] In a possible implementation, the classification module is used to: The fused feature vector is subjected to feature analysis by a classifier to obtain a state classification result output by the classifier.
[0011] In a possible implementation, the classification module is used to: Based on the status classification result output by the classifier and the environment information of the security device, a status classification result of the security device is determined.
[0012] In a possible implementation manner, the device further includes: A feedback module, used to feed back the status classification result of the safety device through target prompt information; The target prompt information includes at least one of the following: text information, image information and audio information.
[0013] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the state identification method of any of the above-mentioned security devices.
[0014] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement any of the above-mentioned methods for identifying the state of a security device.
[0015] On the other hand, a computer program product is provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor to implement any security device status identification method provided in the various optional implementations described above.
[0016] The technical solution provided by this application may have the following beneficial effects: The state recognition method of the security device provided in the embodiment of the present application obtains the device image and device description information of the security device, extracts the image feature vector of the device image and the text feature vector of the device description information, performs feature fusion on the image feature vector and the text feature vector to obtain a fused feature vector, so as to determine the state classification result of the security device based on the fused feature vector, that is, to determine whether the security device is processing a normal state or an abnormal state; through the above method, the multimodal information of the security device can be integrated to extract the fused feature vector, thereby improving the comprehensiveness of feature acquisition, and performing state recognition based on the fused feature vector, thereby improving the accuracy of state recognition of the security device.
[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0019] Figure 1 A flow chart showing a method for identifying a state of a security device provided by an exemplary embodiment of the present application is shown; Figure 2 A flow chart showing a method for identifying a state of a security device provided by an exemplary embodiment of the present application is shown; Figure 3 A block diagram of a state identification device for a security device provided by an exemplary embodiment of the present application is shown; Figure 4 A structural block diagram of a computer device shown in an exemplary embodiment of the present application is shown; Figure 5 A structural block diagram of a computer device shown in an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION
[0020] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0021] Figure 1 A flowchart of a method for identifying the state of a security device provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device, and the computer device can be implemented as a server or a terminal, such as Figure 1 As shown, the state identification method of the safety device may include the following steps: Step 110: Acquire a device image of the security device and device description information of the security device. Safety equipment refers to equipment used to prevent, detect, respond to and mitigate potential risks and hazards, and is intended to protect the safety of people, property and the environment. In principle, safety equipment may include but is not limited to fire extinguishers, smoke detectors, hard hats, safety belts, protective glasses, etc.
[0022] In an embodiment of the present application, a computer device may obtain a device image of a security device through an image acquisition device, such as a camera, a camcorder, or a still camera, etc. The image may be a photograph taken or a video frame captured from a video, and the present application does not impose any restrictions on this.
[0023] The device description information of the security device may be a text description of the security device, and the device description information may include but is not limited to information such as the model, specifications, and appearance of the security device; optionally, the device description information may be structured text information to facilitate semantic understanding and feature extraction.
[0024] Step 120: extract image features from the device image to obtain an image feature vector of the device image.
[0025] Image feature extraction refers to the process of extracting meaningful information (i.e., image features, such as color, texture, edge, shape, etc.) from an image through computer vision technology, and converting the extracted image features into a vector of a specific dimension, i.e., an image feature vector, which is used to characterize each image feature of the device image. In an embodiment of the present application, a computer device can perform image feature extraction through an image feature extraction model to obtain an image feature vector of the device image.
[0026] Step 130 , performing feature fusion on the image feature vector and the text feature vector generated based on the device description information to obtain a fused feature vector.
[0027] The text feature vector is used to represent the semantic information of the device description information; in a possible implementation, the computer device can realize feature fusion of the image feature vector and the text feature vector through a feature fusion model, and the feature fusion model can be a multimodal machine learning model.
[0028] In one possible implementation, the computer device may input the image feature vector and the device description information into a feature fusion model through different input branches respectively, the feature fusion model performs text feature extraction on the device description information to obtain a text feature vector of the device description information, and then performs feature fusion on the text feature vector and the image feature vector; or, in another possible implementation, the computer may first preprocess the device description information (for example, perform text feature extraction through a text feature extraction model) to obtain a text feature vector of the device description information, and then input the image feature vector and the text feature vector into the feature fusion model through different branches, perform feature fusion on the text feature vector and the image feature vector to obtain a fused feature vector.
[0029] In an embodiment of the present application, when fusing an image feature vector and a text feature vector, the image feature vector and the text feature vector can be matched to extract key features common to the image and text, and the extracted image key feature vector and text key feature vector are feature fused to obtain a fused feature vector.
[0030] Step 140, determining a state classification result of the safety device based on the fused feature vector; the state classification result of the safety device is used to indicate whether the safety device is in a normal state or an abnormal state.
[0031] The computer device can analyze and classify the fused feature vector to determine the current status of the security device.
[0032] In summary, the state recognition method of the security device provided in the embodiment of the present application obtains the device image and device description information of the security device, extracts the image feature vector of the device image and the text feature vector of the device description information, and performs feature fusion on the image feature vector and the text feature vector to obtain a fused feature vector, so as to determine the state classification result of the security device based on the fused feature vector, that is, to determine whether the security device is processing a normal state or an abnormal state; through the above method, the multimodal information of the security device can be integrated to extract the fused feature vector, thereby improving the comprehensiveness of feature acquisition, and performing state recognition based on the fused feature vector, thereby improving the accuracy of state recognition of the security device.
[0033] In an embodiment of the present application, a computer device can implement feature extraction and feature fusion processes through a machine learning model; traditional deep learning models require a large amount of labeled data for training, and place high demands on the acquisition and labeling of sample data during the training process, and require a large amount of computing resources to support it, and have high requirements on hardware devices, making it difficult to deploy in a resource-constrained environment; therefore, in order to improve the efficiency of model training, reduce the demand for data labeling, and reduce the computing resources required for model training, the security device identification method provided in an embodiment of the present application can use a pre-trained model, and perform transfer learning or model fine-tuning on the basis of the pre-trained model so that the trained model can be applicable to the status recognition scenario of the security device; wherein a pre-trained model refers to a model that has been trained on a large-scale data set.
[0034] Figure 2 A flowchart of a method for identifying the state of a security device provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device, and the computer device can be implemented as a server or a terminal. As shown in 2, the method for identifying the state of a security device can include the following steps: Step 210: Acquire a device image of a security device and device description information of the security device. Step 220, input the device image into a pre-trained image feature extraction model to obtain an image feature vector of the device image output by the image feature extraction model.
[0035] Since the image quality of the captured device image is affected by environmental information, for example, lighting conditions will affect the contrast and brightness of the captured device image, thereby affecting the image quality, temperature and humidity may affect the physical state of the image capture device itself, thereby affecting the features in the image, and changes in the position of the image capture device may affect the viewing angle and background of the captured device image, thereby affecting feature extraction; based on this, in a possible implementation method, an embodiment of the present application can obtain environmental information of the security device; perform data preprocessing on the device image based on the environmental information to obtain a preprocessed device image; when performing image feature extraction, perform image feature extraction on the preprocessed device image to obtain an image feature vector of the device image.
[0036] In an embodiment of the present application, a computer device can obtain environmental information of a security device through a sensor device. The sensor device may include at least one of the following: an optical sensor, a temperature sensor, a humidity sensor, etc. Correspondingly, the environmental information may include at least one of the following: light intensity, temperature, humidity, etc.
[0037] When image preprocessing is performed on the device image based on environmental information, image preprocessing can be performed according to the environmental information. For example, the exposure time and gain of the image are adjusted according to the light intensity to ensure that the image brightness is moderate, the contrast and brightness of the image are adjusted according to the light intensity to improve the image quality, denoising is performed according to the environmental information to reduce noise interference in the image, unnecessary background interference is removed according to the location information, and safety equipment in the image is highlighted, etc.; further, in the image preprocessing process, image enhancement, image normalization and other operations can also be performed on the image.
[0038] When performing image feature extraction, the computer device extracts image features through an image feature extraction model, and the image feature extraction model is a model obtained after fine-tuning the model through a second training sample set based on a model pre-trained based on a first training sample set; the scale of the first training sample set is larger than the scale of the second training sample set; the second training sample set contains security device sample images and first image labels corresponding to the security device sample images.
[0039] Illustratively, the pre-trained model corresponding to the image extraction model may be a VGG (Visual Geometry Group) model, or a ResNet (Residual Network) model, which may be a model pre-trained on a large-scale data set (i.e., a first training sample set), and may extract common high-level features, such as edges, textures, and shapes of objects in images; thereafter, transfer learning is performed on the pre-trained model based on a second training sample set including security device sample images of security devices and first image labels corresponding to the security device sample images, so that the trained model has better performance in image feature extraction of device images, i.e., is suitable for image feature extraction of device images.
[0040] Image features of the device image are extracted based on the image feature extraction model obtained by transfer learning of the pre-trained model to obtain an image feature vector of the device image; wherein the image feature extraction model may include multiple convolutional layers and pooling layers, wherein the convolutional layer is used to capture local features in the device image, such as edges, textures, etc., and the pooling layer is used to reduce the spatial dimension of the feature map while retaining important information to improve the robustness of the model.
[0041] Step 230, input the image feature vector and the device description information into the pre-trained feature fusion model to obtain the fusion feature output by the feature fusion model.
[0042] Among them, the feature fusion model is a model obtained after fine-tuning the model through the fourth training sample set based on the model pre-trained based on the third training sample set; the scale of the third training sample set is larger than the scale of the fourth training sample set; the fourth training sample set contains image feature vectors and device description information of security device sample images, as well as the second image label corresponding to the security device sample images.
[0043] In an embodiment of the present application, the pre-trained model corresponding to the feature fusion model can be a multimodal pre-trained model, such as a CLIP (Contrastive Language-Image Pre-training) model. The multimodal pre-trained model can be a model pre-trained on a large-scale data set (i.e., the third training sample set), which can match multimodal features to extract key features and perform feature fusion on the extracted multimodal key features; thereafter, transfer learning is performed on the multimodal pre-trained model based on the image feature vector and device description information containing the sample image of the security device, and the fourth training sample set of the second image label corresponding to the sample image of the security device, and the trained model has a better performance in the feature fusion of the device image, that is, it is suitable for the feature fusion of the device image.
[0044] In one possible implementation, the feature fusion model can directly perform feature fusion on the image feature vector and the text feature vector to obtain a fused feature vector; in another possible implementation, the feature fusion model can calculate the similarity between the image feature vector and the text feature vector, determine the feature vectors with a similarity higher than a similarity threshold as key features and extract them, and then perform feature fusion on the extracted multimodal key features to obtain a fused feature vector; the construction method based on the feature fusion model can be based on different feature fusion processes, which is not limited by the present application.
[0045] Optionally, the image label in the sample image may be known information corresponding to the sample image. For example, the image label may be the area where the device body in the image is located, the image quality score, the image type, etc. Correspondingly, during the model training process, a fully connected layer may be added after the pre-trained model or the multimodal pre-trained model for classification or regression tasks, so that the output prediction information and the known information may be combined to calculate the loss function, and then the parameters of the pre-trained model may be updated. After the training completion conditions are met, the trained pre-trained model is determined as an image feature extraction model, and the trained multimodal pre-trained model is determined as a feature fusion model; wherein, the security device sample images in the second training sample set and the fourth training sample set may be the same or different, and the types of image labels may be the same or different.
[0046] Step 240: Perform feature analysis on the fused feature vector through a classifier to obtain a state classification result output by the classifier.
[0047] The status classification result of the safety device is used to indicate whether the safety device is in a normal state or an abnormal state.
[0048] The classifier may be pre-trained based on a training data set with labeled state classification results, and may determine the state classification of the safety device based on an input fusion feature vector; the classifier may be implemented as a support vector machine (SVM), logistic regression, softmax regression, and the like.
[0049] In a possible implementation, the computer device may also comprehensively determine the status classification result of the security device based on the use environment of the security device indicated by the environmental information and the status classification structure of the security device identified by the model. This process may be implemented as follows: Based on the status classification result output by the classifier and the environment information of the security device, a status classification result of the security device is determined.
[0050] In one possible implementation, a computer device may compare environmental information of a security device with standard environmental information corresponding to the security device, where the standard environmental information is used to indicate an environment that ensures that the security device is in a normal state; when the difference between the environmental information of the security device and the standard environmental information is greater than a difference threshold, it is determined that the current security device is in an abnormal working environment, and at this time it can be determined that the security device is in an abnormal state.
[0051] When the status classification result output by the classifier indicates that the safety device is in a normal state, and the environmental information of the safety device indicates that the safety device is in a normal state, the status classification result of the safety device is determined to be a normal state; when either the status classification result output by the classifier or the status classification result indicated by the environmental information of the safety device indicates that the safety device is in an abnormal state, the status classification result of the safety device is determined to be an abnormal state.
[0052] Furthermore, after determining the status classification result of the security device, the computer device may also feed back the status classification result of the security device through target prompt information; The target prompt information includes at least one of the following: text information, image information and audio information.
[0053] Illustratively, the computer device may display text information or image information corresponding to the status classification result through a display screen, or play audio information corresponding to the status classification result through a player.
[0054] In summary, the state recognition method of the security device provided in the embodiment of the present application obtains the device image and device description information of the security device, extracts the image feature vector of the device image and the text feature vector of the device description information, and performs feature fusion on the image feature vector and the text feature vector to obtain a fused feature vector, so as to determine the state classification result of the security device based on the fused feature vector, that is, to determine whether the security device is processing a normal state or an abnormal state; through the above method, the multimodal information of the security device can be integrated to extract the fused feature vector, thereby improving the comprehensiveness of feature acquisition, and performing state recognition based on the fused feature vector, thereby improving the accuracy of state recognition of the security device.
[0055] In addition, by extracting image features and performing feature fusion through pre-trained machine learning models, the accuracy of feature extraction can be improved, thereby improving the accuracy of state recognition of security equipment; by applying pre-trained models to the field of state recognition of security equipment through transfer learning, the dependence on large amounts of labeled data can be reduced, the efficiency of model training can be improved, and the consumption of computing resources can be reduced, so that it can be better applied in resource-constrained environments, thereby improving its applicability in resource-constrained environments.
[0056] Taking a fire extinguisher as an example, the following schematically illustrates the state recognition process of the safety device: In the process of state recognition of a fire extinguisher, a fire extinguisher image is obtained through a camera, and environmental information of the fire extinguisher is recorded through a sensor; after preprocessing the fire extinguisher image, the image feature extraction model (such as VGG) is input into a trained image feature extraction model to extract image features and generate an image feature vector of the fire extinguisher image; the text description information of the fire extinguisher and the image feature vector are input into a trained feature fusion model (such as CLIP) to perform feature fusion of the image feature vector and the text feature vector to obtain a fused feature vector; the fused feature vector is input into a classifier to classify the state of the fire extinguisher based on the fused feature vector to determine whether the fire extinguisher is in a normal state, and the classification result can be displayed on a display screen, or can also be fed back to the user through a sound prompt.
[0057] Assume that the text description of a fire extinguisher is: "5 kg portable ABC dry powder fire extinguisher, red shell, clean nozzle without blockage, pressure gauge pointer in the green area." Through image feature processing and multimodal feature fusion and classification, the computer device can identify the status of the fire extinguisher as follows: Normal state: the fire extinguisher looks intact, the pressure is normal, the nozzle is unobstructed, and the position is correct. Or, if there is a problem with the fire extinguisher, the system can identify: Abnormal state: the fire extinguisher shell is damaged, the pressure gauge pointer is in the red area, and the nozzle is blocked.
[0058] Figure 3 A block diagram of a state identification device for a security device provided by an exemplary embodiment of the present application is shown. The device can perform the following steps: Figure 1 or Figure 2 All or part of the steps of the embodiment shown, such as Figure 3 As shown, the device comprises: A first information acquisition module 310 is used to acquire a device image of a security device and device description information of the security device; A feature extraction module 320 is used to extract image features from the device image to obtain an image feature vector of the device image; A feature fusion module 330 is used to perform feature fusion on the image feature vector and the text feature vector generated based on the device description information to obtain a fused feature vector; The classification module 340 is used to determine a state classification result of the safety device based on the fused feature vector; the state classification result of the safety device is used to indicate whether the safety device is in a normal state or an abnormal state.
[0059] In a possible implementation manner, the device further includes: A second information acquisition module, used to acquire environmental information of the security device; A preprocessing module, used for performing image preprocessing on the device image based on the environmental information to obtain a preprocessed device image; The feature extraction module 320 is used to extract image features from the preprocessed device image to obtain an image feature vector of the device image.
[0060] In a possible implementation, the feature extraction module 320 is used to: Inputting the device image into a pre-trained image feature extraction model to obtain an image feature vector of the device image output by the image feature extraction model; Among them, the image feature extraction model is a model obtained after fine-tuning the model through the second training sample set based on the model pre-trained based on the first training sample set; the scale of the first training sample set is larger than the scale of the second training sample set; the second training sample set contains security device sample images and the first image labels corresponding to the security device sample images.
[0061] In a possible implementation, the feature fusion module 330 is used to: Inputting the image feature vector and the device description information into a pre-trained feature fusion model to obtain fusion features output by the feature fusion model; Among them, the feature fusion model is a model obtained after fine-tuning the model through the fourth training sample set based on the model pre-trained based on the third training sample set; the scale of the third training sample set is larger than the scale of the fourth training sample set; the fourth training sample set contains image feature vectors and device description information of security device sample images, as well as the second image label corresponding to the security device sample image.
[0062] In a possible implementation, the classification module 340 is used to: The fused feature vector is subjected to feature analysis by a classifier to obtain a state classification result output by the classifier.
[0063] In a possible implementation, the classification module 340 is used to: Based on the status classification result output by the classifier and the environment information of the security device, a status classification result of the security device is determined.
[0064] In a possible implementation manner, the device further includes: A feedback module, used to feed back the status classification result of the safety device through target prompt information; The target prompt information includes at least one of the following: text information, image information and audio information.
[0065] To summarize, the state recognition device for a security device provided in an embodiment of the present application obtains a device image and device description information of the security device, extracts an image feature vector of the device image and a text feature vector of the device description information, performs feature fusion on the image feature vector and the text feature vector, and obtains a fused feature vector, so as to determine the state classification result of the security device based on the fused feature vector, that is, to determine whether the security device is processing a normal state or an abnormal state; through the above-mentioned device, the multimodal information of the security device can be integrated to extract the fused feature vector, thereby improving the comprehensiveness of feature acquisition, and performing state recognition based on the fused feature vector, thereby improving the accuracy of state recognition of the security device.
[0066] Figure 4 The structural block diagram of a computer device 400 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as the server in the above-mentioned solution of the present application. The computer device 400 includes a central processing unit (CPU) 401, a system memory 404 including a random access memory (RAM) 402 and a read-only memory (ROM) 403, and a system bus 405 connecting the system memory 404 and the central processing unit 401. The computer device 400 also includes a large-capacity storage device 406 for storing an operating system 409, an application program 410 and other program modules 411.
[0067] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 404 and mass storage device 406 can be collectively referred to as memory.
[0068] According to various embodiments of the present disclosure, the computer device 400 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 400 can be connected to the network 408 through the network interface unit 407 connected to the system bus 405, or the network interface unit 407 can be used to connect to other types of networks or remote computer systems (not shown).
[0069] The memory also includes at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is stored in the memory. The central processing unit 401 implements all or part of the steps in the state identification method of the security device shown in the above-mentioned embodiments by executing the at least one instruction, at least one program, code set or instruction set.
[0070] Figure 5 The following is a block diagram of a computer device 500 according to an exemplary embodiment of the present application. The computer device 500 may be implemented as the above-mentioned terminal, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, etc. The computer device 500 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal, etc.
[0071] Typically, the computer device 500 includes a processor 501 and a memory 502 .
[0072] In some embodiments, the computer device 500 may also optionally include: a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502 and the peripheral device interface 503 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 503 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507 and a power supply 508.
[0073] In some embodiments, the computer device 500 further includes one or more sensors 509 , including but not limited to: an acceleration sensor 510 , a gyroscope sensor 511 , a pressure sensor 512 , an optical sensor 513 , and a proximity sensor 514 .
[0074] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the computer device 500, and the computer device 500 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.
[0075] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement all or part of the steps in the above-mentioned security device state identification method. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.
[0076] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor Figure 1 or Figure 2 All or part of the steps of the method for identifying the state of a security device shown in any embodiment.
[0077] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.
[0078] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for identifying the state of a security device, characterized in that: The method comprises: Acquire a device image of a security device and device description information of the security device; Extracting image features from the device image to obtain an image feature vector of the device image; Performing feature fusion on the image feature vector and the text feature vector generated based on the device description information to obtain a fused feature vector; A state classification result of the safety device is determined based on the fused feature vector; the state classification result of the safety device is used to indicate whether the safety device is in a normal state or an abnormal state.
2. The method according to claim 1, characterized in that The method further comprises: Acquiring environmental information of the security device; Performing image preprocessing on the device image based on the environmental information to obtain a preprocessed device image; The step of extracting image features from the device image to obtain an image feature vector of the device image includes: Image feature extraction is performed on the preprocessed device image to obtain an image feature vector of the device image.
3. The method according to claim 1 or 2, characterized in that: The step of extracting image features from the device image to obtain an image feature vector of the device image includes: Inputting the device image into a pre-trained image feature extraction model to obtain an image feature vector of the device image output by the image feature extraction model; Among them, the image feature extraction model is a model obtained after fine-tuning the model through the second training sample set based on the model pre-trained based on the first training sample set; the scale of the first training sample set is larger than the scale of the second training sample set; the second training sample set contains security device sample images and the first image labels corresponding to the security device sample images.
4. The method according to claim 1, characterized in that: The step of fusing the image feature vector and the text feature vector generated based on the device description information to obtain a fused feature vector includes: Inputting the image feature vector and the device description information into a pre-trained feature fusion model to obtain fusion features output by the feature fusion model; Among them, the feature fusion model is a model obtained after fine-tuning the model through the fourth training sample set based on the model pre-trained based on the third training sample set; the scale of the third training sample set is larger than the scale of the fourth training sample set; the fourth training sample set contains image feature vectors and device description information of security device sample images, as well as the second image label corresponding to the security device sample image.
5. The method according to claim 1, characterized in that The determining the state classification result of the security device based on the fused feature vector includes: The fused feature vector is subjected to feature analysis by a classifier to obtain a state classification result output by the classifier.
6. The method according to claim 5, characterized in that The determining the state classification result of the security device based on the fused feature vector includes: Based on the status classification result output by the classifier and the environment information of the security device, a status classification result of the security device is determined.
7. The method according to claim 1, characterized in that The method further comprises: Feedback of the status classification result of the safety device through target prompt information; The target prompt information includes at least one of the following: text information, image information and audio information.
8. A state recognition device for a safety device, characterized in that: The device comprises: A first information acquisition module, used to acquire a device image of a security device and device description information of the security device; A feature extraction module, used to extract image features from the device image to obtain an image feature vector of the device image; A feature fusion module, used for performing feature fusion on the image feature vector and the text feature vector generated based on the device description information to obtain a fused feature vector; A classification module is used to determine a state classification result of the safety device based on the fused feature vector; the state classification result of the safety device is used to indicate whether the safety device is in a normal state or an abnormal state.
9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the state identification method of the security device according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by a processor to implement the state identification method of a security device as described in any one of claims 1 to 7.