Production field monitoring method and device, electronic equipment and storage medium

Abstract

The invention provides a production field monitoring method and device, electronic equipment and a storage medium, and the method comprises the steps: collecting a plurality of production region images of a first production region at a first time point of each workday through an image sensor disposed on a production field; inputting all the collected production area images into a pre-trained multi-modal large model, and enabling the multi-modal large model to output a production management evaluation result corresponding to each production area image according to a production management rule corresponding to a production field, the production management evaluation result at least comprises a problem description text, a mark image and a rectification measure text corresponding to each second production area which does not conform to the production management rule; and generating a first production management report based on all the obtained production management evaluation results. The system is used for realizing monitoring and intelligent management of a production field, and has high adaptability and reusability.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a monitoring method, device, electronic equipment and storage medium for a production site. Background Art

[0002] Against the backdrop of increasingly fierce competition in the global manufacturing industry, steel production companies are facing huge challenges in improving production efficiency, reducing costs and improving product quality. Lean production management has become an important strategy to enhance corporate competitiveness because it can systematically reduce waste and improve efficiency. Among them, 5S management (sorting, reorganizing, cleaning, cleaning, and literacy) as the basic link of lean production has significantly improved the stability and safety of production by standardizing the working environment and improving the literacy of employees. In the steel industry, the effective implementation of 5S management can reduce equipment failures and optimize production processes, and has therefore been widely used in practice.

[0003] However, traditional 5S management relies heavily on manual execution and supervision. Faced with the complexity and scale of modern steel production, it is difficult to meet the company's requirements for real-time, accuracy and automation. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a production site monitoring method, device, electronic device and storage medium, which are used to realize monitoring and intelligent management of the production site and have high adaptability and reusability.

[0005] In a first aspect, the present invention provides a method for monitoring a production site, the method comprising:

[0006] Through the image sensors set up at the production site, multiple production area images of the first production area are collected at the first time point of each working day; all the collected production area images are input into a pre-trained multimodal large model, so that the multimodal large model outputs the production management evaluation results corresponding to each production area image according to the production management rules corresponding to the production site, and the production management evaluation results at least include the problem description text, marked image and corrective measures text corresponding to each second production area that does not comply with the production management rules; based on all the obtained production management evaluation results, a first production management report is generated.

[0007] In an optional implementation, it also includes: sending the production management report to the management end; and obtaining report confirmation information sent by the management end, and sending the production management report to the target production end corresponding to the second production area.

[0008] In an optional embodiment, it also includes determining, for each working day, whether a second time point after the first time point has been reached; if so, obtaining a current production area image of the second production area through an image sensor provided at the production site; inputting the current production area images of all second production areas into a pre-trained multimodal large model, so that the multimodal large model outputs a production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, so as to determine whether each second production area has been rectified; based on the production management evaluation results corresponding to all second production areas, generating a second production management report, wherein, in the second production management report, the second production areas that have not been rectified are marked in a special manner.

[0009] In an optional embodiment, the multimodal large model is trained and generated in the following manner: a sample data set is generated, each sample data in the sample data set includes a sample production area image marked with an identification box and a question description text, and a question and answer text; the original model is trained with the sample data set, so that the original model outputs the production management evaluation results corresponding to each production area image according to the production management rules corresponding to the production site, and the original model is adjusted to obtain the multimodal large model.

[0010] In an optional implementation, the step of adjusting the original model to obtain a multimodal large model specifically includes:

[0011] For each sample data, the corresponding loss value is calculated to adjust the parameters of the original model. The loss value L is calculated in the following way: total :

[0012]

[0013] Among them, L itc is the contrastive learning loss between image and text, L itm is the matching degree between image and text, L lm is the matching degree value of the text, σ itc , σ itm , σ lm is the corresponding weight coefficient value.

[0014] In an optional implementation, the weight coefficient value is determined by the type of sample data.

[0015] In an optional implementation, the sample data set is generated by:

[0016] Obtain a sample production area image of a preset size, mark the areas that do not comply with production management rules with identification boxes, and mark the type, location, and cause of the problem in the corresponding area with problem description text; for each sample production area image, use the GPT model to generate a sample question and answer text corresponding to the sample production area image according to a preset question and answer text template.

[0017] In a second aspect, the present invention provides a monitoring device for a production site, the device comprising:

[0018] A collection module, used for collecting production area images of a plurality of first production areas at a first time point of each working day through an image sensor arranged at the production site;

[0019] A judgment module is used to input all the collected production area images into a pre-trained multimodal large model, so that the multimodal large model outputs a production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, and the production management evaluation result at least includes a problem description text, a marked image and a rectification measure text corresponding to each second production area that does not comply with the production management rules;

[0020] A generating module is used to generate a first production management report based on all obtained production management evaluation results.

[0021] In a third aspect, the present invention provides an electronic device comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the processor executes the machine-readable instructions to perform the steps of a production site monitoring method as described in any of the aforementioned implementations.

[0022] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a production site monitoring method as described in any one of the aforementioned embodiments are executed.

[0023] The present application provides a production site monitoring method, device, electronic device and storage medium, the method includes collecting multiple production area images of the first production area at the first time point of each working day through an image sensor set up at the production site; inputting all the collected production area images into a pre-trained multimodal large model, so that the multimodal large model outputs the production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, and the production management evaluation result at least includes the problem description text, the marked image and the rectification measure text corresponding to each second production area that does not comply with the production management rules; based on all the obtained production management evaluation results, a first production management report is generated. It is used to realize the monitoring and intelligent management of the production site, and has high adaptability and reusability. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A flowchart of a production site monitoring method provided in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of the structure of a monitoring device for a production site provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] First, the application scenario of the present application is described. The technical solution of the present application is applicable to the management of steel production sites.

[0029] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0030] Figure 1 The following is a flow chart of a production site monitoring method provided in an embodiment of the present application. Figure 1 As shown, a production site monitoring method provided by the present application can be executed by a production management system, specifically comprising:

[0031] S10. Collecting production area images of a plurality of first production areas at a first time point of each working day through an image sensor disposed at the production site.

[0032] The production management system also includes multiple image sensors installed at different points in the production site. Specifically, they can be installed in areas including but not limited to office areas, meeting rooms, spare parts areas, tool storage areas, and control rooms. The image sensors here can be cameras, industrial cameras, and so on.

[0033] The system can set the time for report generation, and can also set the time for production area image acquisition. Exemplarily, the time for the system to generate a production management report every working day can be half an hour before getting off work (such as 5:30). The first time point for the production area image acquisition here can be half an hour before getting off work (5:30), or once at 10 am and once at 3 pm, etc. That is, the first time point can be one time point or multiple time points. When the first time point is multiple time points, the generated production management report can summarize the production problems corresponding to multiple time points, and can also be compared and analyzed.

[0034] S11. Input all the collected production area images into a pre-trained multimodal large model, so that the multimodal large model outputs the production management evaluation results corresponding to each production area image according to the production management rules corresponding to the production site. The production management evaluation results at least include the problem description text, marked image and corrective measures text corresponding to each second production area that does not comply with the production management rules.

[0035] S13. Generate a first production management report based on all obtained production management evaluation results.

[0036] The collected production area images will be automatically input into the trained multimodal model for analysis. The multimodal model combines the pre-imported 5S rules and regulations to conduct a comprehensive assessment of the sorting, rectification, cleaning, and quality of each area. Based on the analysis results, the system automatically generates a 5S production management report, which includes the following parts:

[0037] Location: Indicate the specific location of each monitored production area. Image sensors can be numbered by region, and then the corresponding location can be determined based on the number associated with the production area image, such as a conference room or production workshop.

[0038] Description: Based on the 5S management standard, the actual status of each area is described in detail, including sorting, reorganization, cleaning, hygiene, literacy, and existing safety hazards. For example, the problem description text can be "From the picture, there are several problems: 1. Logo specification issues: Although the yellow and black logos on the handrails exist, the edge lines are not clear enough, and the logos in some areas may be a little worn, affecting the visual effect. 2. Sanitary dust accumulation problem: Dust and dirt can be seen on the surface of the stair steps, especially in the corners and gaps of the steps. According to the 5S management standard, these areas should be kept clean and cleaned regularly to prevent safety hazards.".

[0039] Corrective measures: Based on the identified problems, provide specific corrective suggestions, including items sorting, equipment maintenance, cleaning plan adjustment, etc. For example, the corrective measures may be "1. Repaint: For the parts where the paint has fallen off, especially the stairs and handrails, they should be repainted to ensure that the logo and color are bright and consistent. 2. Cleaning and maintenance: Clean the logo area regularly to prevent dust from blocking the logo, and at the same time extend the service life of the logo." The first production management report also includes an image of the production area marked with an identification box. The identification box here is used to indicate the location of the production problem in the image.

[0040] The present application provides a production site monitoring method that can generate reports in graphic format, which is helpful for 5S management of the production site, ensures the accurate generation of daily management effect reports, and realizes the optimization and improvement of management processes.

[0041] In one embodiment of the present application, the production management report may be sent to the management terminal, and the report confirmation information sent by the management terminal may be obtained, and the production management report may be sent to the target production terminal corresponding to the second production area.

[0042] Each time a production management report is generated, it can be sent to the management end for manual review and confirmation by the administrator, and the report can also be modified. The administrator can further choose to send the production management report to the target production end of the person in charge / employee of the production area with problems, so that they can make corrections.

[0043] For better supervision and management, it is also possible to determine for each working day whether a second time point after the first time point has been reached. If so, the current production area image of the second production area is obtained through the image sensor set up at the production site. The current production area images of all second production areas are input into the pre-trained multimodal large model, so that the multimodal large model outputs the production management evaluation results corresponding to each production area image according to the production management rules corresponding to the production site, so as to determine whether each second production area has been rectified. Based on the production management evaluation results corresponding to all second production areas, a second production management report is generated, wherein the second production areas that have not been rectified are marked in a special way in the second production management report.

[0044] For example, the second production area where production problems are detected can be re-inspected half an hour or one hour later. If the target production area still has the same problems in the second production management assessment result, it means that the production area has not been rectified, and further supervision or even punishment is required.

[0045] In the second production management report here, for the same production area, the results of the two production management assessments can be listed in comparison, and the areas with problems last time can also be marked with identification boxes in the image, so that it can be more intuitive to see whether rectification has occurred.

[0046] In one embodiment of the present application, the system also provides a dialogue function. Managers / persons in charge or employees can actively send pictures and questions to the big model to further explore the details and feasibility of the corrective measures. For example, you can ask questions such as "What are the problems in this area?", "What is the specific definition of the identification specification, how should I rectify it?", etc. The big model will answer based on the preset production management rules and question-and-answer dialogue template.

[0047] In one embodiment of the present application, a multimodal large model can be trained and generated in the following manner:

[0048] (i) Generate a sample data set, where each sample data in the sample data set includes a sample production area image marked with an identification box and a question description text, as well as a question and answer text.

[0049] Specifically, the sample data set can be generated in the following ways:

[0050] Obtain a sample production area image of a preset size, mark the areas that do not comply with production management rules with identification boxes, and mark the type, location, and cause of the problem in the corresponding area with problem description text.

[0051] The sample production area images can be all historical images collected from the production site, and these data will be used to build and optimize the multimodal large model. The sample production area images are acquired through industrial cameras or other image acquisition devices with a resolution of 448×448 pixels. The time span of data collection covers different production cycles and environmental conditions to ensure the comprehensiveness and representativeness of the data. The collected data format is a standard image file format, such as JPEG, PNG, etc.

[0052] These sample production area images will be annotated in detail by on-site personnel, including but not limited to 5S management elements such as sorting, reorganization, and cleaning of the production site. The annotation information will clearly indicate the elements related to 5S management in each image, including but not limited to areas where cleanliness does not meet standards, locations where items are not neatly placed, and potential safety hazards. In addition, the annotator will use rectangles or polygons to frame the areas with problems and record relevant descriptions, such as the type of problem, location, possible causes, etc. These annotated data will serve as an important basis for training models and provide support for accurate recognition and analysis of the models.

[0053] For each sample production area image, the GPT model is used to generate a sample question and answer text corresponding to the sample production area image according to the preset question and answer text template.

[0054] Question and answer text generation can be based on the image annotation information of the production site and the company's 5S management standards, aiming to provide rich training data for the model. First, the annotated image data is combined with the relevant 5S management standards to generate text descriptions related to 5S management. These descriptions cover compliance checks on the production site in terms of organization, rectification, cleaning, hygiene, and literacy.

[0055] Next, we use the method of embedding specific generation prompts to call an API with visual processing capabilities (such as the gpt-4o-mini model based on effect and cost considerations) to generate dialogue data containing image information and text descriptions. The generation process is as follows: the annotated image data and 5S management specification text are input into the gpt-4o-mini model, and the model generates corresponding dialogue content based on the input image and text.

[0056] These conversation data will simulate actual inspection and management scenarios, such as identifying and explaining problems in images and providing improvement suggestions. In the conversation content, the parts involving images will be marked with special tags (such as ) for subsequent processing. The generated dialogue data includes end-to-end joint image and text dialogue (image + text) and pure text dialogue to ensure the model's ability to understand and execute instructions in different scenarios and improve its practicality and reliability in 5S lean production management.

[0057] For example, the prompt of the question and answer text can be as follows:

[0058] "Generate conversation data related to 5S lean production management based on the input image file and annotation file provided, and integrate relevant knowledge of 5S management specifications into the conversation.

[0059] ###Image file and annotation content:

[0060] {input}

[0061] ###Output requirements:

[0062] 1. **Conversation Instance**: The output should contain multiple conversation instances, each corresponding to a set of dialogues related to the image.

[0063] 2. **Conversation structure**: Each conversation instance should include multiple rounds of dialogue between the user and the assistant. The user's questions can involve multiple pictures and text descriptions, and the assistant's answers must be combined with image annotation information and provide suggestions and explanations related to the 5S management standards.

[0064] 3. **Problem description and analysis**: Users may ask specific questions about the images (such as sorting, organizing, sweeping, cleaning, literacy, etc.). The assistant needs to describe the problems based on the images and annotation files, and explain the impact of these problems on the production environment and workflow.

[0065] 4. **Marking box information**: When the user asks to draw a marking box, the assistant's answer should include the location information of the marking box and explain the content and meaning of the marked area through dialogue.

[0066] 5. **5S management knowledge**: The assistant's answer should incorporate knowledge related to 5S management. For example, explain why a certain area needs to be cleaned, how to organize and reorganize items, and the potential risks of not meeting 5S standards.

[0067] 6. **Conversation format**: Conversation data should be output in JSON format, where the user's image reference is ` `Label, and contains the relevant URL link, the location information of the marked box is used` <box>`Tag indicates. "

[0068] (ii) The original model is trained through the sample data set so that the original model outputs the production management evaluation results corresponding to each production area image according to the production management rules corresponding to the production site, and the original model is adjusted to obtain a multimodal large model.

[0069] In a specific embodiment, Qwen-VL-Chat can be selected as the original model of the multimodal large model, and Q-LoRA can be used for fine-tuning. Qwen-VL-Chat has excellent end-to-end joint image and text processing capabilities, can process visual and text information, and has high efficiency and outstanding performance in end-to-end joint image and text tasks. It can accurately process complex image and text information, and has strong adaptability. It can adapt to specific tasks through fine-tuning to improve detection accuracy, mainly with good support for Chinese dialogues. Q-LoRA uses a 4-bit quantization model and paged attention technology, which significantly reduces video memory requirements and is suitable for resource-constrained environments. It can achieve fast and efficient fine-tuning while maintaining model performance, and only needs to store the parameters of the adapter part, which is convenient for model storage and deployment.

[0070] Fine-tuning the model is a key step to improve its performance in 5S lean production management tasks. In order to optimize the model performance and make it more accurate and efficient in handling complex industrial management tasks, a multi-task joint loss function is specially designed.

[0071] The steps to adjust the original model to obtain a multimodal large model include:

[0072] For each sample data, the corresponding loss value is calculated to adjust the parameters of the original model. The loss value L is calculated in the following way: total :

[0073]

[0074] Among them, L itc is the contrastive learning loss between image and text, L itm is the matching degree between image and text, L lm is the matching degree value of the text, σ itc , σ itm , σ lm is the corresponding weight coefficient value.

[0075] In view of the multimodal image-text joint characteristics of the present invention, the model needs to process visual (image) and language (text) information at the same time. In the 5S management task, the model not only needs to accurately identify management problems in the image (such as untidy placement of items, insufficient cleanliness of the area, etc.), but also needs to generate or understand text descriptions related to these problems. Therefore, the loss function design needs to consider both image classification loss and text generation loss to ensure that the model performs well and well on both tasks.

[0076] To this end, the present invention introduces a joint loss function to comprehensively optimize the model performance by combining the losses of image and text tasks.

[0077] Among them, σ itc , σ itm , σ lm is the uncertainty estimate of the corresponding task loss, which can be replaced by the variance. They determine the proportion of each task loss in the total loss. The regularization form prevents the model from minimizing the loss by infinitely increasing these variances, ensuring the stability and generalization ability of the model.

[0078] Specifically, the weight coefficient value can be determined according to the type of sample data.

[0079] In practical applications, the importance of image information and text information in 5S management tasks may vary with specific scenarios. To this end, the present invention designs an adaptive weight adjustment mechanism, using a learnable parameter σ itc 2 , σ itm 2 , σ mm 2 The weights are adjusted dynamically as the training progresses. These parameters are updated as training progresses to reflect the model's uncertainty about each task.

[0080] When the uncertainty of a task is large (σ itc 2 , σ itm 2 , σ lm 2 When the uncertainty is large, the impact of the loss of the task on the total loss is weakened, and the model will focus more on the tasks that it believes it is more certain about. On the contrary, when the uncertainty is small, the model will pay more attention to the loss of the task and ensure accuracy on this task.

[0081] Through this adaptive weighting mechanism, the model can automatically adjust the degree of attention to different tasks and improve its overall performance in complex management scenarios. In addition, by introducing regularization terms, the uncertainty estimate is prevented from increasing infinitely, ensuring the robustness of model training.

[0082] In a specific embodiment of the present application, the production management system can not only analyze image data, but also has the ability to communicate with users, thereby providing more accurate feedback and interaction. The specific construction process is as follows:

[0083] (1) Introduction of 5S rules and regulations in the factory

[0084] The present invention allows the user to import the 5S management rules and regulations of a specific steel plant when the system is initialized or regularly updated. If no specific rules and regulations are imported, the system will use the preset general 5S management standards as the basis for judgment. The imported rules and regulations will serve as a guide for image data analysis and report generation to ensure that the report content meets the actual management requirements of the plant.

[0085] (2) Timing trigger image acquisition

[0086] The system automatically starts at the preset report generation time (for example, half an hour before the end of get off work every day) and collects images of key areas in the factory (including but not limited to office areas, meeting rooms, spare parts areas, tool storage areas and control rooms). The collection equipment covers all important 5S management areas to ensure that the collected data is comprehensive and representative.

[0087] (3) Generate daily 5S management effect report

[0088] The collected image data will be automatically input into the trained model for analysis. The model combines the imported 5S rules and regulations to conduct a comprehensive assessment of the sorting, organization, cleaning, and quality of each area. Based on the analysis results, the system automatically generates a 5S management effect report, which includes the following parts:

[0089] Location: Indicate the specific location of each monitored area.

[0090] Description: Based on the 5S management standard, a detailed description of the actual status of each area is given, including sorting, reorganization, cleaning, hygiene, literacy, and existing safety hazards.

[0091] Corrective measures: Based on the identified problems, specific corrective suggestions are provided, including item sorting, equipment maintenance, cleaning plan adjustment, etc. Users can interact with the system through the dialogue function to further discuss the details and feasibility of the corrective measures.

[0092] Suggestions: Propose further optimization suggestions, such as employee training, process improvement, etc., to improve the overall 5S management level. The system's dialogue function supports users to discuss and optimize these suggestions to ensure the operability and effectiveness of the suggestions.

[0093] Priority: Classify and sort rectification items according to the severity and urgency of the problem, helping managers to prioritize key issues. Users can discuss the priority determination criteria with the system through the dialogue function and adjust them according to actual conditions.

[0094] Evaluation score: Generate a comprehensive 5S management score to reflect the overall management effect of the day.

[0095] (4) Report output and distribution

[0096] The generated report is presented in a combination of pictures and text, with image annotations and text descriptions of the problem areas. The report can be automatically distributed to the terminal or email of the relevant manager through the system. Through the dialogue function, the manager can conduct a final check and confirmation with the system before the report is distributed, and make personalized adjustments as needed.

[0097] (5) Correction follow-up and status update

[0098] The system will assign the person in charge of each rectification item according to the generated report, and require the person in charge to fill in the rectification status in the system. The person in charge can reconfirm the details in the picture through the system dialogue function and compare them with the rules and regulations. The system will provide comprehensive guidance to ensure that various 5S management measures are effectively implemented.

[0099] Through the above process, the present invention realizes the automation and intelligence of 5S management in steel plants. The joint model of the system can not only analyze images, but also has the ability to interact with users in real time, ensuring the accurate generation and effective follow-up of daily management effect reports, and realizing the optimization and improvement of management processes. The rectification results can also be connected to the employee performance system as a reference indicator for assessment, which can also promote better implementation of the system.

[0100] Figure 2 This is a schematic diagram of the structure of a monitoring device for a production site provided in an embodiment of the present application. Figure 2 As shown, based on the same inventive concept, the embodiment of the present application also provides a monitoring device 20 for a production site, the device comprising:

[0101] The acquisition module 210 is used to acquire production area images of a plurality of first production areas at a first time point of each working day through an image sensor arranged at the production site;

[0102] The judgment module 220 is used to input all the collected production area images into a pre-trained multimodal large model, so that the multimodal large model outputs a production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, and the production management evaluation result at least includes a problem description text, a marked image and a rectification measure text corresponding to each second production area that does not comply with the production management rules;

[0103] The generating module 230 is used to generate a first production management report based on all the obtained production management evaluation results.

[0104] In a preferred embodiment, the method further includes: sending the production management report to the management end; and obtaining report confirmation information sent by the management end, and sending the production management report to the target production end corresponding to the second production area.

[0105] In a preferred embodiment, it also includes determining, for each working day, whether a second time point after the first time point has been reached; if so, obtaining the current production area image of the second production area through the image sensor set up at the production site; inputting the current production area images of all the second production areas into a pre-trained multimodal large model, so that the multimodal large model outputs the production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, so as to determine whether each second production area has been rectified; based on the production management evaluation results corresponding to all the second production areas, generating a second production management report, wherein in the second production management report, the second production areas that have not been rectified are marked in a special manner.

[0106] In a preferred embodiment, the multimodal large model is trained and generated in the following manner: a sample data set is generated, each sample data in the sample data set includes a sample production area image marked with an identification box and a question description text, and a question and answer text; the original model is trained with the sample data set, so that the original model outputs the production management evaluation results corresponding to each production area image according to the production management rules corresponding to the production site, and the original model is adjusted to obtain the multimodal large model.

[0107] In a preferred embodiment, the step of adjusting the original model to obtain a multimodal large model specifically includes:

[0108] For each sample data, the corresponding loss value is calculated to adjust the parameters of the original model. The loss value L is calculated in the following way: total :

[0109]

[0110] Among them, L itc is the contrastive learning loss between image and text, L itm is the matching degree between image and text, L lm is the matching degree value of the text, σ itc , σ itm , σ lm is the corresponding weight coefficient value.

[0111] In a preferred embodiment, the weight coefficient value is determined by the type of sample data.

[0112] In a preferred embodiment, the sample data set is generated by:

[0113] Obtain a sample production area image of a preset size, mark the areas that do not comply with production management rules with identification boxes, and mark the type, location, and cause of the problem in the corresponding area with problem description text; for each sample production area image, use the GPT model to generate a sample question and answer text corresponding to the sample production area image according to a preset question and answer text template.

[0114] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown in , the electronic device 300 includes a processor 310 , a memory 320 and a bus 330 .

[0115] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 communicates with the memory 320 through the bus 330. When the machine-readable instructions are executed by the processor 310, the steps of a production site monitoring method in the above method embodiment can be executed. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0116] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a production site monitoring method in the above method embodiment can be executed. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0119] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0121] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM) random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0122] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0123] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.< / box>

Claims

1. A production site monitoring method, characterized in that: The method comprises: By using an image sensor disposed at the production site, a plurality of production area images of the first production area are collected at the first time point of each working day; Input all the collected production area images into a pre-trained multimodal large model, so that the multimodal large model outputs a production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, and the production management evaluation result at least includes a problem description text, a marked image and a rectification measure text corresponding to each second production area that does not comply with the production management rules; Based on all the obtained production management evaluation results, a first production management report is generated.

2. The method according to claim 1, characterized in that Also includes: Sending the production management report to the management terminal; as well as Acquire report confirmation information sent by the management end, and send the production management report to the target production end corresponding to the second production area.

3. The method according to claim 2, characterized in that Also includes: For each working day, determining whether a second time point after the first time point has been reached; If yes, then obtaining the current production area image of the second production area through an image sensor provided at the production site; Input the current production area images of all second production areas into a pre-trained multimodal large model, so that the multimodal large model outputs the production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, so as to determine whether each second production area should be rectified; A second production management report is generated based on the production management evaluation results corresponding to all second production areas, wherein the second production areas that have not been rectified are marked in a special manner in the second production management report.

4. The method according to claim 1, characterized in that: The multimodal large model is trained and generated in the following manner: Generate a sample data set, each sample data in the sample data set includes a sample production area image marked with an identification box and a question description text, and a question and answer text; The original model is trained through the sample data set, so that the original model outputs the production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, and the original model is adjusted to obtain the multimodal large model.

5. The method according to claim 4, characterized in that The step of adjusting the original model to obtain the multimodal large model specifically includes: For each sample data, the corresponding loss value is calculated to adjust the parameters of the original model. The loss value L is calculated in the following way: total : Among them, L utc is the contrastive learning loss between image and text, L itm is the matching degree between image and text, L lm is the matching degree value of the text, σ itc , σ itm , σ lm is the corresponding weight coefficient value.

6. The method according to claim 5, characterized in that The weight coefficient value is determined by the type of sample data.

7. The method according to claim 4, characterized in that Generate a sample dataset by: Obtain a sample production area image of a preset size, mark the area that does not comply with the production management rules with an identification box, and mark the type, location and cause of the problem in the corresponding area with a problem description text; For each sample production area image, the GPT model is used to generate a sample question and answer text corresponding to the sample production area image according to the preset question and answer text template.

8. A monitoring device for a production site, characterized in that: The device comprises: A collection module, used for collecting production area images of a plurality of first production areas at a first time point of each working day through an image sensor arranged at the production site; A judgment module is used to input all the collected production area images into a pre-trained multimodal large model, so that the multimodal large model outputs a production management evaluation result corresponding to each production area image according to the production management rules corresponding to the production site, and the production management evaluation result at least includes a problem description text, a marked image and a rectification measure text corresponding to each second production area that does not comply with the production management rules; A generating module is used to generate a first production management report based on all obtained production management evaluation results.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of a production site monitoring method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the production site monitoring method as claimed in any one of claims 1 to 7 are executed.

Images