6S violation detection method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a 6S violation detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a plurality of to-be-detected images; inputting the plurality of to-be-detected images into a target detection model, and determining a predicted violation image from the plurality of to-be-detected images; inputting the prediction violation image and the target prompt instruction into the first multi-mode large model to obtain a violation detection result output by the first multi-mode large model; wherein the first multi-modal large model is obtained by performing fine adjustment on the initial multi-modal large model based on the 6S violation sample image and the violation label corresponding to the 6S violation sample image, and the violation label is obtained by inputting the 6S violation sample image and the violation positioning prompt word corresponding to the 6S violation sample image into the second multi-modal large model. According to the invention, the detection efficiency of 6S violation detection can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a 6S violation detection method, device, electronic device and storage medium. Background Art

[0002] At present, most factories adopt 6S management to provide employees with an orderly, efficient and safe production environment and improve production efficiency and quality. 6S management includes sorting (Seiri), straightening (Seiton), cleaning (Seiso), cleaning (Seiketsu), literacy (Shitsuke) and safety (Security), which refers to the effective management of production factors such as personnel, machines, materials and methods in the production environment.

[0003] In the prior art, most factories rely on manual inspection when implementing 6S management, or use deep learning algorithms to screen and evaluate images captured by cameras, which are then reviewed by 6S administrators. Manual inspection methods are time-consuming and have low detection efficiency. In addition, due to the influence of subjective judgment, it is difficult to ensure the stability and accuracy of the detection results. The methods that use deep learning algorithms and manual review still rely on high-frequency manual review, which places a heavy workload on 6S administrators. In addition, due to the complexity and diversity of 6S violation types, different classifiers need to be trained through data from different 6S violation types to achieve classification of detection content, and the detection efficiency is low. Therefore, how to improve the detection efficiency of 6S violation detection is an urgent problem that needs to be solved. Summary of the invention

[0004] The present invention provides a 6S violation detection method, device, electronic device and storage medium, which are used to solve the defect of low detection efficiency of 6S violation detection in the prior art.

[0005] The present invention provides a 6S violation detection method, which comprises the following steps.

[0006] Acquire multiple images to be detected; Inputting the plurality of images to be detected into a target detection model, and determining a predicted illegal image from the plurality of images to be detected; Inputting the predicted violation image and the target prompt instruction into a first multimodal large model to obtain a violation detection result output by the first multimodal large model; Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

[0007] According to the 6S violation detection method provided by the present invention, the first multimodal large model is obtained by fine-tuning based on the following steps: Based on the classification information and detection frame information of the 6S violation sample image, determine the violation location prompt word corresponding to the 6S violation sample image; Inputting the 6S violation sample image and the violation location prompt word corresponding to the 6S violation sample image into the second multimodal large model, and obtaining the violation label corresponding to the 6S violation sample image output by the second multimodal large model; Constructing a training data set based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images; The initial multimodal large model is fine-tuned based on the training data set to obtain the first multimodal large model.

[0008] According to the 6S violation detection method provided by the present invention, the determining of the violation location prompt word corresponding to the 6S violation sample image based on the classification information and the detection frame information of the 6S violation sample image includes: Obtaining a prompt word template, and identifying the type and location information of the violation in the prompt word template; The classification information is used to replace the violation type, and the detection frame information is used to replace the location information, so as to obtain the violation location prompt word corresponding to the 6S violation sample image.

[0009] According to the 6S violation detection method provided by the present invention, the initial multimodal large model is fine-tuned based on the training data set to obtain the first multimodal large model, including: The training data set is input into the initial multimodal large model, and the local parameters in the initial multimodal large model are fine-tuned using a low-rank matrix decomposition method to obtain the first multimodal large model.

[0010] According to the 6S violation detection method provided by the present invention, the step of inputting the plurality of images to be detected into a target detection model and determining a predicted violation image from the plurality of images to be detected comprises: Input each of the images to be detected into the target detection model to obtain the recognition accuracy and prediction box corresponding to each of the images to be detected; Determine an image to be detected whose recognition accuracy rate among the multiple images to be detected is greater than a first preset threshold as an abnormal image; A prediction box corresponding to the abnormal image is added to the abnormal image to obtain a predicted violation image.

[0011] According to the 6S violation detection method provided by the present invention, the target prompt instruction is obtained based on the following steps: Determine the difference between 1 and the first preset threshold; When the difference is greater than a second preset threshold, determining the first prompt instruction and the second prompt instruction as the target prompt instruction; When the difference is less than a second preset threshold, determining the second prompt instruction as the target prompt instruction; The first prompt instruction is used to instruct the first multimodal large model to determine whether it belongs to a 6S violation object; the second prompt instruction is used to instruct the first multimodal large model to generate violation description information of the predicted violation image.

[0012] The present invention also provides a 6S violation detection device, comprising the following modules.

[0013] An acquisition module, used for acquiring multiple images to be detected; A first detection module, used for inputting the plurality of images to be detected into a target detection model, and determining a predicted illegal image from the plurality of images to be detected; A second detection module, used for inputting the predicted violation image and the target prompt instruction into the first multimodal large model to obtain a violation detection result output by the first multimodal large model; Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the 6S violation detection method described above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the 6S violation detection method as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned 6S violation detection methods.

[0017] The 6S violation detection method, device, electronic device and storage medium provided by the present invention screen out predicted violation images from multiple images to be detected through a target detection model, and input the predicted violation images and the corresponding target prompt instructions into the first multimodal large model to determine the violation detection results corresponding to the predicted violation images. In the present invention, the 6S violation sample images and the corresponding violation location prompt words are input into the second multimodal large model, and the image features in the 6S violation sample images and the text features in the corresponding violation location prompt words are associated with the second multimodal large model to obtain violation labels, and the initial multimodal large model is fine-tuned according to the 6S violation sample images and the violation labels to improve the detection accuracy of the first multimodal large model after fine-tuning. Different 6S violation types can be detected by the first multimodal large model, reducing the number of models that need to be trained, reducing the subsequent manual review workload, and automatically determining the violation labels in combination with the second multimodal large model to improve the determination efficiency of the training data, thereby improving the detection efficiency of 6S violation detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of a 6S violation detection method provided by an embodiment of the present invention.

[0020] Figure 2 It is a schematic diagram of the training process of the first multimodal large model provided in an embodiment of the present invention.

[0021] Figure 3 It is a structural schematic diagram of a 6S violation detection device provided in an embodiment of the present invention.

[0022] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In view of the problem that the detection efficiency of 6S violation detection in the prior art is low, an embodiment of the present invention provides a 6S violation detection method. Figure 1 6S violation detection method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes the following steps 110 to 130.

[0025] Step 110: Acquire multiple images to be detected.

[0026] It should be noted that the multiple images to be detected can be images of the production environment captured by a camera in the factory. That is, after the camera shoots a video of the production environment, the video of the production environment is extracted to obtain multiple detection images. The multiple images to be detected can reflect the real situation of the production environment in the factory. The number of the cameras can be multiple, and they are deployed at different locations in the factory.

[0027] It should be noted that multiple images to be detected may involve different aspects of 6S management. 6S management includes six aspects: sorting (Seiri), tidying (Seiton), cleaning (Seiso), cleaning (Seiketsu), literacy (Shitsuke) and security (Security). Among them: sorting refers to classifying all items in the production environment into useful and useless items. The purpose of sorting is to clean up all useless items, keep useful items, make space, and maintain a clean production environment. Rectification refers to placing the retained useful items in the specified position, placing them neatly and marking them. The purpose of rectification is to establish a neat production environment, place items in an orderly manner, and reduce the time to find related items. Cleaning refers to cleaning the production environment. The purpose of cleaning is to keep the production environment clean and facilitate the discovery of potential problems and hidden dangers. Cleaning refers to carrying out sorting, tidying and cleaning to the end and institutionalizing them. The purpose of cleaning is to ensure the continuous improvement of the production environment through institutionalization. Literacy refers to requiring employees to strictly abide by and implement regulations. The purpose of literacy is to cultivate employees' self-discipline and teamwork spirit. Safety refers to providing safety education to employees to prevent accidents before they occur. The purpose of safety is to require employees to always pay attention to safety, comply with safety regulations, and prevent accidents from happening.

[0028] The image to be detected may include violations or compliance phenomena in 6S management. For example, in the aspect of sorting in 6S management, the image to be detected may include a messy work area, that is, there are a large number of unnecessary items piled up in the work area, resulting in a small space for activities in the work area, which is not easy to work. It can also include a neat work area. In the aspect of rectification in 6S management, the image to be detected may include unclear markings in the work area, that is, the items are not clearly marked or the markings are blurred and difficult to identify. It can also include tools and equipment that are not returned to their positions, that is, the tools and equipment after use are not placed in the designated position, resulting in low work efficiency. It can also include items placed in an orderly manner. In the aspect of cleaning in 6S management, the image to be detected may include insufficient cleanliness of the work area, that is, there are unclean phenomena such as garbage, dust, and oil in the work area. It can also include regular cleaning of production equipment and tools. In the aspect of cleaning in 6S management, the image to be detected may include a dirty production environment, and regular cleaning and sorting are not carried out according to the cleaning system. In terms of literacy in 6S management, the images to be detected may include irregular employee behavior, such as employees not wearing work clothes or safety helmets, etc. In terms of safety in 6S management, the images to be detected may include potential safety hazards in the production environment, such as exposed wires and damaged equipment in the production environment, etc. It may also include safety measures in place in the production environment, such as setting up safety warning signs and configuring necessary safety equipment in the production environment, etc.

[0029] Step 120: Input multiple images to be detected into the target detection model, and determine the predicted illegal images from the multiple images to be detected.

[0030] Specifically, the target detection model can be a pre-trained YOLO detection model. After each image to be detected is input into the YOLO detection model, it is detected whether there is a violation object in the image to be detected. If there is a violation object, the image to be detected is determined as a predicted violation image. After each image to be detected is detected, the predicted violation images can be preliminarily screened out from multiple images to be detected, thereby reducing the amount of data for subsequent first multimodal large model detection.

[0031] Furthermore, a plurality of images to be detected are input into the target detection model, and a predicted illegal image is determined from the plurality of images to be detected, including: Input each image to be detected into the target detection model to obtain the recognition accuracy and prediction box corresponding to each image to be detected; Determine an image to be detected whose recognition accuracy rate is greater than a first preset threshold among the multiple images to be detected as an abnormal image; Add the prediction box corresponding to the abnormal image to the abnormal image to obtain the predicted violation image.

[0032] For example, taking the target detection model as a pre-trained YOLO detection model, after each image to be detected is input into the YOLO detection model, each image to be detected is subjected to pre-processing operations such as image scaling, cropping or filling to obtain a corresponding pre-processed image. The YOLO detection model extracts features layer by layer on each pre-processed image, and the YOLO detection model predicts multiple bounding boxes, as well as the confidence and category probabilities corresponding to each bounding box. Each bounding box is used to locate the position of the target object, and the confidence is used to characterize the probability of the existence of the target object in each bounding box. The category probability is used to characterize the probability that the target object in the bounding box belongs to the 6S violation category, which includes six categories: sorting violation category, rectification violation category, cleaning violation category, cleaning violation category, literacy violation category and safety violation category. Afterwards, the bounding box with the highest confidence is output as a prediction box, and the category probability corresponding to the prediction box is determined as the recognition accuracy of the prediction box. If the recognition accuracy is less than or equal to the first preset threshold, it indicates that the target object in the prediction box is less likely to be a 6S violation object, that is, the probability that the image to be detected is a normal image is greater. If the recognition accuracy is greater than the first preset threshold, it indicates that the target object in the prediction box is likely to be a 6S violation object, that is, the probability that the image to be detected is a violation image is relatively high. At this time, the image to be detected can be determined as an abnormal image, and after marking the prediction box on the abnormal image, a predicted violation image is obtained, which is convenient for subsequent further confirmation of the predicted violation image.

[0033] It should be noted that the first preset threshold is used as a threshold value for violation judgment in the target detection model and as a lower limit value for judgment. Through the first preset threshold, the image to be detected can be divided into two categories, one is a normal image and the other is an abnormal image. The first preset threshold can be set according to historical experience. For example, the first preset threshold can be 60%, 80%, 90% or 95%, etc., which is not limited in the embodiment of the present invention.

[0034] Step 130: input the predicted violation image and the target prompt instruction into the first multimodal large model to obtain the violation detection result output by the first multimodal large model.

[0035] Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

[0036] Specifically, after determining the predicted violation image with a higher violation risk, the predicted violation image and the target prompt instruction can be input into the first multimodal large model, which is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample image and the corresponding violation label, and the violation label is obtained by identifying the 6S violation sample image and the corresponding violation location prompt word through the second multimodal large model. The first multimodal large model can understand the user's intention through the target prompt instruction, and confirm the violation of the predicted violation image according to the user's intention, thereby determining the violation detection result corresponding to each predicted violation image, that is, reconfirming whether each predicted violation image includes a 6S violation object, and if it does, further describing the 6S violation object in a long text.

[0037] It should be noted that both the first multimodal large model and the second multimodal large model are multimodal large language models (MLLM). Multimodal large language models can use powerful large language models as a harmony engine to perform multimodal tasks, and can process and understand data in multiple modes such as text, images, audio and video. Multimodal large models have been applied in various fields. For example, in the medical field, multimodal large models are used for medical impact analysis and electronic medical record processing. By combining image data such as X-ray film, MRI (Magnetic Resonance Imaging), and CT (Computed Tomography) scans with medical record text, doctors are assisted in making more accurate diagnosis and treatment decisions, thereby improving the quality and efficiency of medical services. For example, in the field of intelligent manufacturing, the appearance image of the equipment captured by the camera and computer vision technology can be used to detect whether the equipment has external wear, cracks or other physical damage. It should be noted that the second multimodal large model can be a general multimodal large language model, such as Tongyi Qianwen or IMP-3B model, and the first multimodal large model is obtained by fine-tuning the general multimodal large language model through 6S violation sample images and violation labels corresponding to 6S violation sample images, so that the first multimodal large model is used for 6S violation detection with higher pertinence and detection accuracy. In addition, the type of the first multimodal large model and the type of the second multimodal large model can be the same or different, and the embodiment of the present invention does not limit this.

[0038] Further, Figure 2 is a schematic diagram of the training process of the first multimodal large model provided in an embodiment of the present invention, such as Figure 2 As shown, the first multimodal large model is fine-tuned based on the following steps: Based on the classification information and detection frame information of the 6S violation sample image, determine the violation location prompt word corresponding to the 6S violation sample image; Inputting the 6S violation sample image and the violation location prompt word corresponding to the 6S violation sample image into the second multimodal large model, and obtaining the violation label corresponding to the 6S violation sample image output by the second multimodal large model; Based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, a training data set is constructed; The initial multimodal large model is fine-tuned based on the training data set to obtain a first multimodal large model.

[0039] Specifically, the 6S violation sample image is data that has been simply annotated and classified. After obtaining the 6S violation sample image, the number of images belonging to different 6S violation categories in the 6S violation sample image can be balanced and adjusted to prevent the second multimodal large model from being biased towards a specific category due to the category ratio deviation of the sample image when answering. After the balance adjustment, the classification information and the detection frame information can be extracted from the 6S violation sample image, the nature of the violation behavior can be clarified through the classification information, and the location of the violation object can be clarified through the detection frame information, so as to combine and obtain the corresponding violation location prompt word. After that, the 6S violation sample image and the corresponding violation location prompt word are input into the second multimodal large model. The second multimodal large model clarifies the user's intention, the corresponding violation category and the location of the violation object by parsing the violation location prompt word. After that, the image indicated by the location of the violation object in the 6S violation sample image is parsed by using the powerful image understanding ability, and the corresponding violation label is generated by combining the violation category by using the powerful text generation ability. Afterwards, the 6S violation sample images and the corresponding violation labels constitute a training data set, and the initial multimodal large model is fine-tuned through the training data set to obtain the first multimodal large model that has been trained.

[0040] Optionally, after the second multimodal large model outputs the violation labels corresponding to the 6S violation sample images based on the violation location prompt words and the 6S violation sample images, manual screening can be performed to filter out violation labels that do not meet the requirements or are incorrectly identified, and retain high-quality violation labels.

[0041] It should be noted that while constructing the training data set, it is also necessary to construct a test data set based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images. The ratio of the test data set to the training data set is preferably 2:8, which is not limited in this embodiment of the present invention.

[0042] In the embodiment of the present invention, the second multimodal large model is used to identify the images after simple manual annotation, and the violation labels of different 6S violation categories are output, which greatly reduces the requirements and costs of manual annotation, and makes the constructed training data set more accurate and high-quality, thereby improving the recognition accuracy of the subsequent first multimodal large model. In addition, the first multimodal large model can learn the association between 6S violation sample images of different 6S violation categories and the corresponding violation labels, avoiding the training of different classifiers to classify the corresponding 6S violation categories, facilitating model management and improving model training efficiency.

[0043] Furthermore, based on the classification information and detection frame information of the 6S violation sample image, the violation location prompt word corresponding to the 6S violation sample image is determined, including: Obtain a prompt word template, and identify the violation type and location information in the prompt word template; The classification information is used to replace the violation type, and the detection box information is used to replace the location information, so as to obtain the violation location prompt words corresponding to the 6S violation sample image.

[0044] Specifically, after obtaining the 6S violation sample image, obtain a prompt word template, which includes the violation type and location information. For example, the prompt word template can be: Please note that the "violation type" phenomenon is detected in the area formed by the "location information" of the image. After that, extract the manually annotated classification information and detection frame information from the 6S violation sample image, and replace the "violation type" in the prompt word template with the classification information, and replace the "location information" in the prompt word template with the detection frame information, and obtain the violation location prompt word of the 6S violation sample image, and the violation location prompt word is: Please note that the "classification information" phenomenon is detected in the area formed by the "detection frame information" of the image. For example, taking the classification information as items are placed in a messy manner, and the detection box information includes the upper left corner coordinates of (50,50) and the lower right corner coordinates of (200,200), after replacement in the prompt word template, the obtained violation positioning prompt word is: Please note that in the area formed by the "upper left corner coordinates of (50,50) and the lower right corner coordinates of (200,200)" of the image, the "items are placed in a messy manner" phenomenon is detected.

[0045] It should be noted that the detection frame information includes coordinate information of the detection frame, and the coordinate information may include the coordinate information of the upper left corner and the lower right corner of the detection frame, or include the coordinate information of the upper right corner and the lower left corner of the detection frame, or include the coordinate information of the four corners of the detection frame. The embodiment of the present invention is not limited to this.

[0046] Furthermore, the initial multimodal large model is fine-tuned based on the training data set to obtain a first multimodal large model, including: The training data set is input into the initial multimodal large model, and the local parameters in the initial multimodal large model are fine-tuned using the low-rank matrix decomposition method to obtain the first multimodal large model.

[0047] Specifically, before training the initial multimodal large model, a low-rank matrix decomposition method is used to perform low-rank decomposition on the weight matrix of the initial multimodal large model to obtain two smaller weight sub-matrices to reduce the complexity and content requirements of the model while maintaining the performance of the model. After that, the training data set is input into the initial multimodal large model, and the predicted output result of the model is obtained through forward propagation, and the difference between the predicted output result and the violation label is determined. The two smaller weight sub-matrices after the decomposition of the initial multimodal large model are updated by calculating the loss function for back propagation, without updating other parameters in the initial multimodal large model. Repeat the above steps until the updated initial multimodal large model reaches stability on the validation set or the iterative training rounds reach the preset rounds, and the first multimodal large model that has been trained is obtained.

[0048] Furthermore, the target prompt instruction is obtained based on the following steps: Determine a difference between 1 and a first preset threshold; When the difference is greater than a second preset threshold, determining the first prompt instruction and the second prompt instruction as target prompt instructions; When the difference is less than a second preset threshold, determining the second prompt instruction as the target prompt instruction; The first prompt instruction is used to instruct the first multimodal large model to determine whether it belongs to a 6S violation object; the second prompt instruction is used to instruct the first multimodal large model to generate violation description information of the predicted violation image.

[0049] Specifically, before the predicted illegal image is input into the trained first multimodal large model, in order to make the first multimodal large model understand the user's intention, it is also necessary to input the target prompt instruction into the first multimodal large model. At this time, the difference between 1 and the first preset threshold can be calculated. Since the value range of the first preset threshold is [0,1], the difference is a value greater than or equal to 0. After that, the difference is compared with the second preset threshold. If the difference is greater than the second preset threshold, it indicates that the threshold value set by the target detection model when detecting the predicted illegal image is low, and the predicted illegal image obtained is more likely to be misdetected. At this time, both the first prompt instruction and the second prompt instruction can be used as the target prompt instruction, and the target prompt instruction and the predicted violation image are input into the first multimodal large model together, instructing the first multimodal large model to detect again whether there is a 6S violation object in the predicted violation image. If there is, continue to generate the violation description information of the predicted violation image. The violation description information can be the first multimodal large model identifying the position, shape, layout, color and other violation visual information of the object in the prediction box in the predicted violation image, and converting the identified violation visual information into accurate and clear long text description information, so as to obtain the violation detection result including the violation judgment result and the violation description information. If the difference is less than the second preset threshold, it indicates that the threshold value set by the target detection model when detecting the predicted violation image is high, and the possibility of false detection of the obtained predicted violation image is low. At this time, only the second prompt instruction can be used as the target prompt instruction, that is, to confirm again that there is a 6S violation object in the predicted violation image. After the target prompt instruction and the predicted violation image are input into the first multimodal large model, the first multimodal large model is instructed to perform a long text description of the 6S violation object in the prediction box in the predicted violation image, so as to obtain the violation detection result including only the violation description information.

[0050] It should be noted that the second preset threshold is used as a threshold value for misdetection judgment in the target detection model and as a lower limit value for judgment. Through the second preset position, the predicted violation image can be divided into two categories, one is a misdetected image and the other is a non-misdetected image. The second preset threshold can be set according to historical experience. For example, the second preset threshold can be 30%, 25%, 20%, etc., which is not limited in the embodiment of the present invention.

[0051] The 6S violation detection method provided by the embodiment of the present invention screens out predicted violation images from multiple images to be detected through a target detection model, and inputs the predicted violation images and the corresponding target prompt instructions into the first multimodal large model to determine the violation detection results corresponding to the predicted violation images. In the present invention, the 6S violation sample images and the corresponding violation location prompt words are input into the second multimodal large model, and the image features in the 6S violation sample images and the text features in the corresponding violation location prompt words are associated with the second multimodal large model to obtain violation labels, and the initial multimodal large model is fine-tuned according to the 6S violation sample images and the violation labels to improve the detection accuracy of the first multimodal large model after fine-tuning. Different 6S violation types can be detected by the first multimodal large model, reducing the number of models that need to be trained, reducing the subsequent manual review workload, and automatically determining the violation labels in combination with the second multimodal large model to improve the determination efficiency of the training data, thereby improving the detection efficiency of 6S violation detection.

[0052] The 6S violation detection device provided by the present invention is described below. The 6S violation detection device described below and the 6S violation detection method described above can be referenced to each other.

[0053] The embodiment of the present invention also provides a 6S violation detection device, Figure 3 6S violation detection device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the 6S violation detection device 300 includes: an acquisition module 310 , a first detection module 320 and a second detection module 330 .

[0054] The acquisition module 310 is used to acquire multiple images to be detected.

[0055] The first detection module 320 is used to input multiple images to be detected into the target detection model, and determine the predicted illegal image from the multiple images to be detected.

[0056] The second detection module 330 is used to input the predicted violation image and the target prompt instruction into the first multimodal large model to obtain the violation detection result output by the first multimodal large model.

[0057] Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

[0058] The 6S violation detection device provided by the embodiment of the present invention screens out predicted violation images from multiple images to be detected through a target detection model, and inputs the predicted violation images and the corresponding target prompt instructions into the first multimodal large model to determine the violation detection results corresponding to the predicted violation images. In the present invention, the 6S violation sample images and the corresponding violation location prompt words are input into the second multimodal large model, and the image features in the 6S violation sample images and the text features in the corresponding violation location prompt words are associated with the second multimodal large model to obtain violation labels, and the initial multimodal large model is fine-tuned according to the 6S violation sample images and the violation labels to improve the detection accuracy of the first multimodal large model after fine-tuning. Different 6S violation types can be detected by the first multimodal large model, reducing the number of models that need to be trained, reducing the subsequent manual review workload, and automatically determining the violation labels in combination with the second multimodal large model to improve the determination efficiency of the training data, thereby improving the detection efficiency of 6S violation detection.

[0059] Optionally, the 6S violation detection device 300 further includes a training module, which is specifically used to: Based on the classification information and detection frame information of the 6S violation sample image, determine the violation location prompt word corresponding to the 6S violation sample image; Inputting the 6S violation sample image and the violation location prompt word corresponding to the 6S violation sample image into the second multimodal large model, and obtaining the violation label corresponding to the 6S violation sample image output by the second multimodal large model; Constructing a training data set based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images; The initial multimodal large model is fine-tuned based on the training data set to obtain the first multimodal large model.

[0060] Optionally, the training module is specifically used for: Obtaining a prompt word template, and identifying the type and location information of the violation in the prompt word template; The classification information is used to replace the violation type, and the detection frame information is used to replace the location information, so as to obtain the violation location prompt word corresponding to the 6S violation sample image.

[0061] Optionally, the training module is specifically used for: The training data set is input into the initial multimodal large model, and the local parameters in the initial multimodal large model are fine-tuned using a low-rank matrix decomposition method to obtain the first multimodal large model.

[0062] Optionally, the first detection module 320 is specifically configured to: Input each of the images to be detected into the target detection model to obtain the recognition accuracy and prediction box corresponding to each of the images to be detected; Determine an image to be detected whose recognition accuracy rate among the multiple images to be detected is greater than a first preset threshold as an abnormal image; A prediction box corresponding to the abnormal image is added to the abnormal image to obtain a predicted violation image.

[0063] Optionally, the second detection module 330 is specifically configured to: Determine the difference between 1 and the first preset threshold; When the difference is greater than a second preset threshold, determining the first prompt instruction and the second prompt instruction as the target prompt instruction; When the difference is less than a second preset threshold, determining the second prompt instruction as the target prompt instruction; The first prompt instruction is used to instruct the first multimodal large model to determine whether it belongs to a 6S violation object; the second prompt instruction is used to instruct the first multimodal large model to generate violation description information of the predicted violation image.

[0064] Figure 4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the 6S violation detection method, which includes: Acquire multiple images to be detected; Inputting the plurality of images to be detected into a target detection model, and determining a predicted illegal image from the plurality of images to be detected; Inputting the predicted violation image and the target prompt instruction into a first multimodal large model to obtain a violation detection result output by the first multimodal large model; Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

[0065] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0066] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the 6S violation detection method provided by the above methods, the method includes: Acquire multiple images to be detected; Inputting the plurality of images to be detected into a target detection model, and determining a predicted illegal image from the plurality of images to be detected; Inputting the predicted violation image and the target prompt instruction into a first multimodal large model to obtain a violation detection result output by the first multimodal large model; Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

[0067] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the 6S violation detection method provided by the above methods, the method comprising: Acquire multiple images to be detected; Inputting the plurality of images to be detected into a target detection model, and determining a predicted illegal image from the plurality of images to be detected; Inputting the predicted violation image and the target prompt instruction into a first multimodal large model to obtain a violation detection result output by the first multimodal large model; Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

[0068] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0069] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 6S violation detection method, characterized in that: include: Acquire multiple images to be detected; Inputting the plurality of images to be detected into a target detection model, and determining a predicted illegal image from the plurality of images to be detected; Inputting the predicted violation image and the target prompt instruction into a first multimodal large model to obtain a violation detection result output by the first multimodal large model; Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

2. The 6S violation detection method according to claim 1, characterized in that: The first multimodal large model is fine-tuned based on the following steps: Based on the classification information and detection frame information of the 6S violation sample image, determine the violation location prompt word corresponding to the 6S violation sample image; Inputting the 6S violation sample image and the violation location prompt word corresponding to the 6S violation sample image into the second multimodal large model, and obtaining the violation label corresponding to the 6S violation sample image output by the second multimodal large model; Constructing a training data set based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images; The initial multimodal large model is fine-tuned based on the training data set to obtain the first multimodal large model.

3. The 6S violation detection method according to claim 2, characterized in that: The determining, based on the classification information and the detection frame information of the 6S violation sample image, the violation location prompt word corresponding to the 6S violation sample image includes: Obtaining a prompt word template, and identifying the type and location information of the violation in the prompt word template; The classification information is used to replace the violation type, and the detection frame information is used to replace the location information, so as to obtain the violation location prompt word corresponding to the 6S violation sample image.

4. The 6S violation detection method according to claim 2, characterized in that: The fine-tuning of the initial multimodal large model based on the training data set to obtain the first multimodal large model includes: The training data set is input into the initial multimodal large model, and the local parameters in the initial multimodal large model are fine-tuned using a low-rank matrix decomposition method to obtain the first multimodal large model.

5. The 6S violation detection method according to any one of claims 1 to 4, characterized in that: The step of inputting the plurality of images to be detected into a target detection model and determining a predicted illegal image from the plurality of images to be detected comprises: Input each of the images to be detected into the target detection model to obtain the recognition accuracy and prediction box corresponding to each of the images to be detected; Determine an image to be detected whose recognition accuracy rate among the multiple images to be detected is greater than a first preset threshold as an abnormal image; A prediction box corresponding to the abnormal image is added to the abnormal image to obtain a predicted violation image.

6. The 6S violation detection method according to claim 5, characterized in that: The target prompt instruction is obtained based on the following steps: Determine the difference between 1 and the first preset threshold; When the difference is greater than a second preset threshold, determining the first prompt instruction and the second prompt instruction as the target prompt instruction; When the difference is less than a second preset threshold, determining the second prompt instruction as the target prompt instruction; The first prompt instruction is used to instruct the first multimodal large model to determine whether it belongs to a 6S violation object; the second prompt instruction is used to instruct the first multimodal large model to generate violation description information of the predicted violation image.

7. A 6S violation detection device, characterized in that: include: An acquisition module, used for acquiring multiple images to be detected; A first detection module, used for inputting the plurality of images to be detected into a target detection model, and determining a predicted illegal image from the plurality of images to be detected; A second detection module, used for inputting the predicted violation image and the target prompt instruction into the first multimodal large model to obtain a violation detection result output by the first multimodal large model; Among them, the first multimodal large model is obtained by fine-tuning the initial multimodal large model based on the 6S violation sample images and the violation labels corresponding to the 6S violation sample images, and the violation labels are obtained by inputting the 6S violation sample images and the violation location prompt words corresponding to the 6S violation sample images into the second multimodal large model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the 6S violation detection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the 6S violation detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the 6S violation detection method according to any one of claims 1 to 6 is implemented.

Images