Establishment method and device of inspection task, equipment, storage medium and program product

Abstract

The invention discloses a routing inspection task establishment method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a scene image set of a target site; according to the scene image set, the inspection category of each scene image and an inspection module corresponding to each inspection category are determined, and the inspection module comprises at least one inspection object; obtaining the position information of the first user at the corresponding position of each scene image, at least one piece of guide information and the detection information of each inspection object, wherein different pieces of guide information correspond to different inspection objects; according to the at least one piece of position information, the inspection module corresponding to each piece of position information, the detection information of at least one inspection object in each inspection module and the guiding information corresponding to each inspection object, an inspection execution model of the target site is established, and the inspection execution model comprises an inspection task of at least one inspection object; different polling objects correspond to different position information and different guiding information, and the polling task is generated by detection information of the polling objects.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and in particular, relates to a method, device, equipment, storage medium and program product for establishing an inspection task. Background Art

[0002] Inspection work is crucial to ensure the normal operation of industrial systems (including production lines, large-scale production equipment, and important pipelines). This work can efficiently maintain and repair industrial equipment and promptly eliminate problems that may exist in long-term operation, thereby fundamentally improving the safety, efficiency, and stability of industrial equipment. In recent years, with the gradual promotion of mixed reality (MR) technology in industrial applications, the smart inspection system in which inspectors wear mixed reality headsets or glasses has gradually been applied in industrial sites. However, the current MR inspection application needs to be customized according to different production tasks. The customization process is cumbersome and requires not only professional program developers to participate, but also engineers with rich production experience and rich inspection experience to participate in planning, including writing scripts, setting standards, and managing processes. In addition, it is necessary to test and debug at the industrial site, resulting in low development efficiency of MR inspection applications. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, device, storage medium and program product for establishing an inspection task, which can improve the development efficiency of the inspection execution model of the target site.

[0004] In a first aspect, an embodiment of the present application provides a method for establishing an inspection task, the method comprising:

[0005] Acquire a scene image set of a target site, wherein the scene image set includes at least one scene image;

[0006] Determine, according to the scene image set, an inspection category of each scene image and an inspection module corresponding to each inspection category, wherein the inspection module includes at least one inspection object;

[0007] Acquire location information of the first user at a location corresponding to each of the scene images, at least one guide information, and detection information of each of the inspection objects, where different guide information corresponds to different inspection objects;

[0008] An inspection execution model of the target site is established based on at least one of the location information, the inspection modules corresponding to each of the location information, the detection information of at least one inspection object in each of the inspection modules, and the guidance information corresponding to each of the inspection objects. The inspection execution model includes an inspection task for at least one of the inspection objects, different inspection objects correspond to different location information and different guidance information, and the inspection task is generated from the detection information of the inspection object.

[0009] In a second aspect, an embodiment of the present application provides a device for establishing an inspection task, the device comprising:

[0010] A first acquisition module, used to acquire a scene image set of a target site, wherein the scene image set includes at least one scene image;

[0011] A first determination module, configured to determine, according to the scene image set, an inspection category of each scene image and an inspection module corresponding to each inspection category, wherein the inspection module includes at least one inspection object;

[0012] A second acquisition module is used to acquire the position information of the first user at the position corresponding to each of the scene images, at least one guide information and the detection information of each of the inspection objects, where different guide information corresponds to different inspection objects;

[0013] A module is established, which is used to establish an inspection execution model of the target site based on at least one of the location information, the inspection modules corresponding to each of the location information, the detection information of at least one inspection object in each of the inspection modules, and the guidance information corresponding to each of the inspection objects, wherein the inspection execution model includes an inspection task of at least one of the inspection objects, different inspection objects correspond to different location information and different guidance information, and the inspection task is generated by the detection information of the inspection object.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for establishing an inspection task as described in any one of the above items is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for establishing an inspection task as described in any one of the above items is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for establishing an inspection task as described in any one of the above.

[0017] The inspection task establishment method, apparatus, equipment, storage medium and program product of the embodiments of the present application can, when a scene image set of a target site is obtained, determine the inspection category of each scene image and the inspection module corresponding to each inspection category based on the scene image set, the inspection module including at least one inspection object, and the scene image set including at least one scene image; obtain position information of a first user at a position corresponding to each scene image, at least one guidance information and detection information of each inspection object, different guidance information corresponding to different inspection objects; establish an inspection execution model for the target site based on at least one position information, the inspection module corresponding to each position information, the detection information of at least one inspection object in each inspection module and the guidance information corresponding to each inspection object, the inspection execution model including the inspection task of at least one inspection object, different inspection objects corresponding to different position information and different guidance information, and the inspection task is generated from the detection information of the inspection object. In this way, in the embodiment of the present application, the inspection category of each scene image and the inspection module corresponding to each inspection category can be identified based on the scene image set of the target site, and then combined with the various location information and various guidance information collected by the inspection personnel when inspecting the target site, the inspection execution model of the target site is finally established. In this way, the developers only need to develop the inspection modules corresponding to each inspection category, and the inspection personnel do not have to participate in the development planning work. The decoupling of development work and inspection tasks reduces the difficulty of developing the inspection execution model of the target site, thereby improving the development efficiency of the inspection execution model of the target site. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 It is a flowchart of a method for establishing an inspection task provided in an embodiment of the present application;

[0020] Figure 2 This is an architectural diagram of a system for establishing inspection tasks provided in an embodiment of the present application;

[0021] Figure 3 It is a structural diagram of a device for establishing an inspection task provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0024] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0025] Inspection work is crucial to ensure the normal operation of industrial systems (including production lines, large-scale production equipment, and important pipelines). This work can efficiently maintain and repair industrial equipment and promptly eliminate problems that may exist in long-term operation, thereby fundamentally improving the safety, efficiency, and stability of industrial equipment. In recent years, with the gradual promotion of mixed reality (MR) technology in industrial applications, the smart inspection system in which inspectors wear mixed reality headsets or glasses has gradually been applied in industrial sites. However, the current MR inspection application needs to be customized according to different production tasks. The customization process is cumbersome and requires not only professional program developers to participate, but also engineers with rich production experience and rich inspection experience to participate in planning, including writing scripts, setting standards, and managing processes. In addition, it is necessary to test and debug at the industrial site, resulting in low development efficiency of MR inspection applications.

[0026] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment, storage medium and program product for establishing an inspection task. The following first introduces the method for establishing an inspection task provided by the embodiments of the present application.

[0027] Figure 1 FIG. 1 is a flow chart showing a method for establishing an inspection task provided by an embodiment of the present application. Figure 1As shown, a method for establishing an inspection task may include the following steps S101 to S104:

[0028] S101, obtaining a scene image set of a target site, where the scene image set includes at least one scene image;

[0029] S102, determining, according to the scene image set, an inspection category of each scene image and an inspection module corresponding to each inspection category, wherein the inspection module includes at least one inspection object;

[0030] S103, obtaining location information of the first user at a location corresponding to each scene image, at least one guide information, and detection information of each inspection object, where different guide information corresponds to different inspection objects;

[0031] S104. Establish an inspection execution model for the target site based on at least one location information, the inspection modules corresponding to each location information, the detection information of at least one inspection object in each inspection module, and the guidance information corresponding to each inspection object. The inspection execution model includes an inspection task for at least one inspection object. Different inspection objects correspond to different location information and different guidance information. The inspection task is generated from the detection information of the inspection object.

[0032] The method for establishing an inspection task in an embodiment of the present application can, when a scene image set of a target site is obtained, determine the inspection category of each scene image and the inspection module corresponding to each inspection category based on the scene image set, the inspection module including at least one inspection object, and the scene image set including at least one scene image; obtain position information of a first user at a position corresponding to each scene image, at least one guidance information and detection information of each inspection object, different guidance information corresponding to different inspection objects; establish an inspection execution model for the target site based on at least one position information, the inspection module corresponding to each position information, detection information of at least one inspection object in each inspection module and guidance information corresponding to each inspection object, the inspection execution model including an inspection task for at least one inspection object, different inspection objects corresponding to different position information and different guidance information, and the inspection task is generated from the detection information of the inspection object. In this way, in the embodiment of the present application, the inspection category of each scene image and the inspection module corresponding to each inspection category can be identified based on the scene image set of the target site, and then combined with the various location information and various guidance information collected by the inspection personnel when inspecting the target site, the inspection execution model of the target site is finally established. In this way, the developers only need to develop the inspection modules corresponding to each inspection category, and the inspection personnel do not have to participate in the development planning work. The decoupling of development work and inspection tasks reduces the difficulty of developing the inspection execution model of the target site, thereby improving the development efficiency of the inspection execution model of the target site.

[0033] In S101, the target site may be an industrial site where mixed reality MR intelligent inspection is required.

[0034] The above scene image set may be a set of scene images, which together depict the environment of the target site. The above scene image set may include at least one scene image, and different scene images may be used to describe different scenes in the target site. Accordingly, different scene images correspond to different location information.

[0035] The above-mentioned acquisition of the scene image set of the target site may, illustratively, be performed by collecting each scene image of the target site through an MR head-mounted display device or MR glasses, and aggregating the scene images in the target site to obtain the scene image set.

[0036] In S102, the inspection categories may include industrial categories, production line types, equipment categories, etc. Different scene images may belong to the same inspection category or different inspection categories. There is a one-to-one correspondence between the inspection categories and the inspection modules.

[0037] The inspection module may include at least one inspection object. The inspection object may refer to an entity or system that needs to be inspected, monitored or evaluated during the inspection process. The inspection object may be various types of equipment, facilities, systems, environments, etc., depending on the purpose and scope of the inspection.

[0038] The above-mentioned method of determining the inspection category of each scene image and the inspection module corresponding to each inspection category based on the scene image set can, exemplarily, be based on multiple inspection categories in a preset initial model to identify the inspection category of each scene image in the scene image set, where the initial model also includes inspection modules of multiple inspection categories; among the inspection modules of multiple inspection categories, the inspection modules corresponding to the inspection category of each scene image are matched.

[0039] In S103, the first user may be, for example, an inspection engineer in the development phase.

[0040] The above-mentioned location information may refer to the specific location data of the first user in the physical space. The location information generally includes coordinate data such as the user's longitude, latitude, and altitude (elevation).

[0041] The above-mentioned guidance information may be information for the first user to guide or convey to find the inspection object. Exemplarily, the guidance information may include sight direction, head direction, finger direction and body direction.

[0042] The detection information of the inspection object mentioned above may refer to various data, status, performance, health status and other information about the inspection object collected during the inspection process, which can help evaluate the status, performance and safety of the inspection object. The detection information of the inspection object, for example, may be to obtain the pressure gauge reading; if the pressure gauge reading is 0.5MPa, the reading is normal, and you can go to the next inspection point; if the pressure gauge reading exceeds 0.7MPa, the reading is abnormal, and you need to go to device A immediately to check the flow meter reading.

[0043] The above-mentioned acquisition of the position information of the first user at the corresponding position of each scene image, at least one guidance information and the detection information of each inspection object can, for example, be that the inspection engineer collects the position information of the corresponding position of each scene image through an MR head-mounted display device or MR glasses, and then collects the guidance information such as the line of sight direction, head direction, finger direction and body direction to guide the inspection engineer to find the instrument when inspecting at the position, and finally collects the detection information about the instrument input by the inspection engineer through voice.

[0044] In S104, the inspection execution model may include an inspection task for at least one inspection object, different inspection objects correspond to different position information and different guidance information, and the inspection task is generated from the detection information of the inspection object.

[0045] The above-mentioned inspection execution model of the target site is established based on at least one position information, the inspection modules corresponding to each position information, the detection information of at least one inspection object in each inspection module and the guidance information corresponding to each inspection object, and the inspection execution model of the target site is established based on at least one position information, the inspection modules corresponding to each position information, the detection information of at least one inspection object in each inspection module and at least one of the line of sight direction, head direction, finger direction and body direction in the guidance information corresponding to each inspection object.

[0046] As an implementation of the present application, in order to improve inspection efficiency and quality, after S104, the method may further include:

[0047] When the second user inspects the target site, obtaining the second user's location information and guidance information;

[0048] Inputting the location information and guidance information of the second user into the inspection execution model, determining a target inspection task, where the target inspection task is an inspection task of an inspection object corresponding to the location information and guidance information of the second user in the inspection execution model, and the target inspection task includes target detection information;

[0049] Acquire interaction information of the second user performing an inspection action indicated by the target detection information on the target inspection object;

[0050] The inspection result of the target inspection task is determined based on the interaction information and the target detection information. The inspection result is used to indicate whether the target inspection task is qualified.

[0051] The second user may be an inspector in the inspection execution phase. In the embodiment of the present application, the first user and the second user may be the same user or different users.

[0052] The above-mentioned target inspection task is an inspection task of the inspection object corresponding to the location information and guidance information of the second user in the inspection execution model, and the target inspection task includes target detection information.

[0053] In an embodiment of the present application, during the inspection execution stage, the location information and guidance information of the second user can be input into the inspection execution model to generate a target inspection task, and the second user can be guided to perform the inspection actions indicated by the target inspection task, and finally the inspection results can be evaluated in real time. The above process can supervise, guide and assist inspection personnel in carrying out inspection work, thereby improving inspection efficiency and quality.

[0054] In some embodiments, the target detection information may include normal operating conditions and treatment opinions for the normal operating conditions and abnormal operating conditions and treatment opinions for the abnormal operating conditions;

[0055] The inspection result of the target inspection task is determined based on the interaction information and the target detection information, which may specifically include:

[0056] In the case where the interactive information is a normal working condition, determining that the inspection result of the target inspection task is qualified, and sending the handling opinion of the normal working condition to the second user;

[0057] In the case where the interactive information is an abnormal working condition, the inspection result of the target inspection task is determined to be unqualified, and the handling opinion of the abnormal working condition is sent to the second user.

[0058] The target detection information may include normal working conditions and treatment opinions for normal working conditions and abnormal working conditions and treatment opinions for abnormal working conditions. For example, the normal working condition may be a pressure gauge reading of 0.5MPa, and the corresponding treatment opinion for the normal working condition may be that the reading is normal and you can go to the next inspection point; the abnormal working condition may be a pressure gauge reading exceeding 0.7Mpa, and the corresponding treatment opinion for the abnormal working condition may be that you need to go to device A immediately to check the flow meter reading.

[0059] In an embodiment of the present application, it is possible to determine in real time whether the inspection result is qualified based on the interaction information and target detection information, and immediately send the corresponding processing opinions to the second user, so that the user can immediately obtain the inspection results and processing suggestions, reduce waiting time, and improve inspection efficiency.

[0060] As another implementation of the present application, in order to further improve the accuracy of the inspection task execution, the above method may also include:

[0061] If the interaction information is an abnormal condition, generating a selection confirmation message and sending it to the terminal device of the second user, the selection confirmation message is used to confirm whether the interaction information is correct;

[0062] Receiving a confirmation operation sent by a terminal device of the second user;

[0063] If the confirmation operation indication interaction information is incorrect, a prompt message is generated, and the prompt message is used to prompt the second user to perform the target inspection task;

[0064] When the confirmation operation indication interaction information is confirmed to be correct, the target detection information of the target inspection task in the inspection execution model is adjusted according to the interaction information.

[0065] The above selection confirmation information can be used to confirm whether the interaction information is correct.

[0066] The terminal device of the second user mentioned above may be, for example, an MR head-mounted display device or MR glasses.

[0067] The above prompt information can be used to prompt the second user to perform the target inspection task.

[0068] In an embodiment of the present application, when the interaction information is an abnormal condition, selection confirmation information can be generated and sent to the terminal device of the second user to confirm the correctness of the interaction information, and the target detection information in the inspection execution model can be adjusted according to the confirmation result, thereby further improving the accuracy of the inspection task execution.

[0069] In some embodiments, the above S102 may specifically include:

[0070] Based on multiple inspection categories in a preset initial model, identifying the inspection category of each scene image in the scene image set, the initial model also includes inspection modules of multiple inspection categories;

[0071] Among the inspection modules of multiple inspection categories, the inspection module corresponding to the inspection category of each scene image is matched.

[0072] The above initial model may include inspection modules of multiple inspection categories, and different inspection types correspond to different inspection modules.

[0073] In the embodiment of the present application, since the initial model includes multiple inspection categories and inspection modules corresponding to each inspection category, the inspection category of each scene image in the scene image set is identified, and the inspection module corresponding to the inspection category of each scene image is matched. This can standardize the operating process, which not only reduces the development workload, but also reduces human errors, ensures the accuracy and reliability of the inspection results, and can also add or modify the inspection modules according to actual needs to adapt to different inspection needs and scenarios.

[0074] In some embodiments, the guidance information may include sight direction, head direction, finger direction and body direction. The above S104 may specifically include:

[0075] An inspection execution model of the target site is established based on at least one location information, the inspection modules corresponding to each location information, the detection information of at least one inspection object in each inspection module, and at least one of the line of sight direction, head direction, finger direction and body direction in the guidance information corresponding to each inspection object.

[0076] The above-mentioned line of sight may refer to judging the object or area that others are focusing on or paying attention to by observing the direction of their eyes.

[0077] The head pointing mentioned above may refer to judging the object or area that another person is focusing on or paying attention to by observing the direction of his / her head.

[0078] The above-mentioned finger pointing may refer to judging the object or area that another person is pointing or guiding by observing the finger pointing of another person.

[0079] The above-mentioned body orientation may refer to judging the object or area that others are focusing on or paying attention to by observing their body orientation.

[0080] In an embodiment of the present application, an inspection execution model of the target site can be accurately established based on at least one location information, the inspection modules corresponding to each location information, the detection information of at least one inspection object in each inspection module, and at least one of the line of sight direction, head direction, finger direction and body direction in the guidance information corresponding to each inspection object.

[0081] In order to facilitate the understanding of the method for establishing the inspection task in the embodiment of the present application, the actual application process of the method for establishing the inspection task is described as follows:

[0082] This application combines the MR system with generative AI technology and embeds MR technology into the training of large AI models to solve the technical problems that MR inspection applications on the market are currently monotonous, require heavy customized development, have long development cycles and high costs.

[0083] In the system to which the embodiments of the present application are applicable, the back end adopts a multimodal large model for industrial segments (equivalent to the above-mentioned inspection execution model), and the front end adopts natural human-computer interaction inputs including voice, gaze point, gesture, spatial position, body movement, etc., to collect multimodal large model training data on the one hand, and to automatically generate an application input interface on the other hand. Finally, the MR head-mounted display device is used as the terminal hardware carrier to collect the user's voice, gaze point, gesture, spatial position and other information, which is transmitted to the multimodal large model at the back end through the network to generate corresponding inspection tasks. After receiving the MR head-mounted display device, it is displayed at the corresponding time and space.

[0084] The whole system can be divided into two parts: the first is to train the big model using the training corpus collected by MR, which can quickly accumulate a large amount of actual training corpus and improve the training efficiency of the big model in the subdivided field; the second is to generate inspection tasks in real time based on the trained multimodal big model according to the comprehensive feedback data of the corresponding MR headset. The data of the real-time inspection process can also optimize the multimodal data of the inspection personnel to improve the accuracy of the inspection tasks. The specific plan is introduced as follows:

[0085] 1. Training phase:

[0086] Step 1: Use the MR head display to obtain the scene image sequence information (equivalent to the above-mentioned scene image set) of the site to be inspected (equivalent to the above-mentioned target site), and the initial large model is trained using the image sequence information to obtain the category sequence information to be inspected.

[0087] The initial large model has many inspection modules corresponding to the inspection categories (industry categories, production line types, equipment categories). When the image information obtained by the MR headset (real-time acquisition by the MR headset) is transmitted to the initial large model, the initial large model will be trained based on the image sequence information to identify the information belonging to the inspection category and the inspection module information of the image, and map this information to the initial large model to obtain the inspection category, inspection module and corresponding sequence information.

[0088] Step 2: Obtain the spatial positioning information (equivalent to the above-mentioned position information) and interaction information (including the above-mentioned guidance information and detection information) collected by the MR device, the interaction information includes but is not limited to: eye tracking information (equivalent to the line of sight in the above-mentioned guidance information), body movement information (equivalent to the body orientation in the above-mentioned guidance information), voice information (equivalent to the detection information of the above-mentioned voice input) and gesture information (equivalent to the finger pointing in the above-mentioned guidance information), based on the interaction information and the parameters of the inspection module, train each inspection module to obtain the proposed detection information (i.e., the above-mentioned detection information) containing the inspection sequence.

[0089] Among them, the parameters of each inspection module may include the inspector's position, the focus point (equivalent to the above-mentioned inspection object), the operation execution parameter (equivalent to the above-mentioned detection information), etc.

[0090] By using the spatial positioning information of MR, the user's location in the scene can be determined, and then the corresponding inspection module can be found.

[0091] MR is used to collect user interaction information. During model training, the interaction information and the parameters of the corresponding inspection module can be mapped to obtain the intended detection information under the inspection module through training.

[0092] Among them, the gaze point of the inspector in the corresponding inspection module is calculated using the eye information and spatial positioning information.

[0093] Step 3: Fit the category sequence information to be inspected and the intended detection information of each inspection module, and train the inspection execution model.

[0094] The following is a practical example to illustrate the inspector's operation information during the initial large model training phase:

[0095] Suppose an inspection engineer wears an MR headset, stands in front of an equipment cabinet (position and head orientation), looks at a meter on a cabinet, points his finger at the meter, and says (voice): "The pressure gauge reads 0.5 MPa, which is normal. You can go to the next inspection point." He also says: "There is an abnormality. If the pressure gauge reading exceeds 0.7 MPa, you need to go to equipment A immediately to check the flow meter reading." The corpus information collected here is shown in Table 1 below.

[0096] Table 1:

[0097]

[0098] In addition, if the factory has a digital twin, the factory digital twin can be three-dimensionally rendered in an MR headset. Experienced inspectors can wear an MR headset and patrol the virtual digital factory, avoiding the need to collect corpus at the industrial site. Without interrupting normal industrial production, training corpus can be obtained quickly and at low cost.

[0099] Compared with traditional MR inspection applications, this inspection training solution can adaptively train the inspection execution model suitable for the scenario by training the parameters obtained from the first operation of the MR headset.

[0100] This training process involves: adaptation of input and output data dimensions; alignment of multiple input data, including alignment in time, space and data scale; setting of training objective function (loss function) and its association with inspection application.

[0101] 2. Inspection execution stage

[0102] Step 1: After the inspection starts, the spatial position information of the MR headset is obtained, and the spatial position information is input into the inspection execution model. The inspection module and the intended detection information (equivalent to the above-mentioned target detection information) that should appear at this time are calculated and determined, and transmitted to the MR headset to generate the display information to be inspected.

[0103] Step 2: Obtain the interactive information collected by the MR head display at this time, and input it into the inspection execution model, and determine the inspection result based on the mapping relationship in the inspection execution model;

[0104] Step 3: When the inspection result is determined to be qualified, the inspection execution model determines the next inspection category information based on the category sequence information to be inspected, and sends it to the MR headset for display.

[0105] Step 4: After all inspection modules have been completed, the data collected by the MR headset is sent to the inspection execution model for training again to obtain an optimized inspection execution model.

[0106] When the inspector's operation information is identified as inaccurate or out of error range by the big model during the inspection process, the big model will generate comparison information between the two, send it to the MR head display for display, and generate a selection confirmation message to let the inspector choose one of the operation information. If the selected operation information is the information previously trained and collected by the big model, the operation information for re-executing the inspection module will be generated, and the inspector will be asked to perform an operation check on the inspection module again; if the selected operation information is the current operation information, it will be fed back to the big model to train the inspection module again, and a prompt message will be generated for the inspection module to be re-tested, so that the inspector can re-test the inspection module after a certain period of time.

[0107] This application uses the actual sensor data of the MR terminal on site, based on the previously generated large model and MR spatial positioning information, to generate inspection task information suitable for the corresponding inspection module, and instruct the inspection engineer to complete the inspection task. And when it is found that there is a difference between the operation and the data of the large model, it can generate comparison information to let the inspector confirm whether the inspection module needs to be retrained.

[0108] For example, an inspection engineer wears an MR headset to enter an industrial site. At this time, all the sensing functions of the MR headset are turned on, and real-time transmission prompts are sent to the backend large model server. The response of the large model server is displayed in the MR headset in the form of voice broadcast or text or other visual annotations (arrows, annotation boxes, PDF documents, pictures, etc.), as shown in Table 2 below.

[0109] Table 2:

[0110]

[0111]

[0112] Large model response:

[0113] 1. Voice or text: "Turn your head and look at device B"

[0114] 2. Visual annotation: "Arrow points to device B"

[0115] 3. Voice or text: "Go to device B now"

[0116] 4. Visual annotation: "The annotation box highlights the indicator light L and switch S on device B"

[0117] 5. Voice or text: "Check indicator light L. If it shows green, it is normal. If it shows yellow, please turn on switch S. If it shows red, please contact the factory command center immediately."

[0118] like Figure 2 As shown in the figure, during the back and forth between the MR headset and the multimodal large model server, the inspection subtasks are dynamically generated, and the inspection task is finally completed step by step. The real-time data generated by this inspection task can be fed back to the large model for further training in real time to further optimize the large model.

[0119] The present application relates to a generative MR industrial inspection method based on a large model and natural human-computer interaction. The method obtains inspection category information by acquiring a scene image sequence training model through an MR head display, and obtains spatial positioning and interaction information of the MR device, trains an inspection module to obtain information to be detected, and combines the category sequence with the inspection module's information to be detected to train an inspection execution model. During the inspection, the MR head display uses position and interaction information to calculate the display information, results, and next category to be inspected, optimizes the model, further improves and accurately produces inspection results, and improves inspection efficiency.

[0120] Based on the method for establishing the inspection task provided in the above embodiment, the present application also provides a specific implementation of the device for establishing the inspection task. Please refer to the following embodiment.

[0121] like Figure 3 As shown, the inspection task establishment device 300 provided in the embodiment of the present application may include the following modules: a first acquisition module 301 , a first determination module 302 , a second acquisition module 303 and an establishment module 304 .

[0122] A first acquisition module 301 is used to acquire a scene image set of a target site, where the scene image set includes at least one scene image;

[0123] A first determination module 302 is used to determine the inspection category of each scene image and the inspection module corresponding to each inspection category according to the scene image set, and the inspection module includes at least one inspection object;

[0124] The second acquisition module 303 is used to obtain the position information of the first user at the corresponding position of each scene image, at least one guide information and the detection information of each inspection object, where different guide information corresponds to different inspection objects;

[0125] Establish module 304, which is used to establish an inspection execution model of the target site based on at least one location information, the inspection modules corresponding to each location information, the detection information of at least one inspection object in each inspection module and the guidance information corresponding to each inspection object. The inspection execution model includes an inspection task of at least one inspection object. Different inspection objects correspond to different location information and different guidance information. The inspection task is generated by the detection information of the inspection object.

[0126] The device for establishing the inspection task in the embodiment of the present application can, after obtaining a set of scene images of the target site, determine the inspection category of each scene image and the inspection module corresponding to each inspection category based on the scene image set, the inspection module including at least one inspection object, and the scene image set including at least one scene image; obtain the position information of the first user at the corresponding position of each scene image, at least one guidance information and the detection information of each inspection object, different guidance information corresponds to different inspection objects; establish an inspection execution model of the target site based on at least one position information, the inspection module corresponding to each position information, the detection information of at least one inspection object in each inspection module and the guidance information corresponding to each inspection object, the inspection execution model including the inspection task of at least one inspection object, different inspection objects correspond to different position information and different guidance information, and the inspection task is generated from the detection information of the inspection object. In this way, in the embodiment of the present application, the inspection category of each scene image and the inspection module corresponding to each inspection category can be identified based on the scene image set of the target site, and then combined with the various location information and various guidance information collected by the inspection personnel when inspecting the target site, the inspection execution model of the target site is finally established. In this way, the developers only need to develop the inspection modules corresponding to each inspection category, and the inspection personnel do not have to participate in the development planning work. The decoupling of development work and inspection tasks reduces the difficulty of developing the inspection execution model of the target site, thereby improving the development efficiency of the inspection execution model of the target site.

[0127] As an implementation of the present application, in order to improve inspection efficiency and quality, the above-mentioned device 300 may further include:

[0128] A third acquisition module is used to acquire the location information and guidance information of the second user when the second user inspects the target site;

[0129] A second determination module is used to input the location information and guidance information of the second user into the inspection execution model to determine a target inspection task, where the target inspection task is an inspection task of an inspection object corresponding to the location information and guidance information of the second user in the inspection execution model, and the target inspection task includes target detection information;

[0130] A fourth acquisition module is used to acquire interaction information of the second user performing an inspection action indicated by the target detection information on the target inspection object;

[0131] The third determination module is used to determine the inspection result of the target inspection task according to the interaction information and the target detection information, and the inspection result is used to indicate whether the target inspection task is qualified.

[0132] In some embodiments, the target detection information may include normal operating conditions and treatment opinions for the normal operating conditions and abnormal operating conditions and treatment opinions for the abnormal operating conditions;

[0133] The third determination module may specifically include:

[0134] A first determination unit is used to determine that the inspection result of the target inspection task is qualified when the interaction information is a normal working condition, and send a processing opinion of the normal working condition to the second user;

[0135] The second determination unit is used to determine that the inspection result of the target inspection task is unqualified when the interaction information is an abnormal working condition, and send the handling opinion of the abnormal working condition to the second user.

[0136] As another implementation of the present application, in order to further improve the accuracy of the inspection task execution, the above-mentioned device 300 may also include:

[0137] A first generating module, used for generating selection confirmation information and sending it to the terminal device of the second user when the interaction information is in an abnormal condition, wherein the selection confirmation information is used to confirm whether the interaction information is correct;

[0138] A receiving module, used to receive a confirmation operation sent by a terminal device of the second user;

[0139] The second generating module generates prompt information when the confirmation operation indication interaction information is confirmed to be incorrect, and the prompt information is used to prompt the second user to perform the target inspection task;

[0140] The adjustment module is used to adjust the target detection information of the target inspection task in the inspection execution model according to the interactive information when the confirmation operation indication interactive information is confirmed to be correct.

[0141] In some embodiments, the first determining module 302 may specifically include:

[0142] An identification unit, used to identify the inspection category of each scene image in the scene image set based on multiple inspection categories in a preset initial model, wherein the initial model also includes inspection modules of multiple inspection categories;

[0143] The matching unit is used to match the inspection modules of multiple inspection categories with the inspection modules corresponding to the inspection category of each scene image.

[0144] In some embodiments, the above-mentioned guidance information may include line of sight direction, head direction, finger direction and body direction. The above-mentioned establishment module 304 is specifically used to establish an inspection execution model of the target site based on at least one position information, the inspection modules corresponding to each position information, the detection information of at least one inspection object in each inspection module, and at least one of the line of sight direction, head direction, finger direction and body direction in the guidance information corresponding to each inspection object.

[0145] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0146] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0147] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0148] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0149] In certain embodiments, the memory 402 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0150] The processor 401 reads and executes the computer program instructions stored in the memory 402 to implement any one of the inspection task establishment methods in the above embodiments.

[0151] In one example, the electronic device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0152] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0153] Bus 410 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0154] The electronic device can execute the method for establishing the inspection task in the embodiment of the present application, thereby realizing the combination Figure 1 and Figure 3 The invention describes a method and device for establishing an inspection task.

[0155] In addition, in combination with the method for establishing the inspection task in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the inspection task establishment methods in the above embodiment is implemented.

[0156] In combination with the method for establishing the inspection task in the above embodiments, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes any one of the methods for establishing the inspection task as described above.

[0157] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0158] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0159] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0160] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0161] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for establishing an inspection task, characterized in that: include: Acquire a scene image set of a target site, wherein the scene image set includes at least one scene image; Determine, according to the scene image set, an inspection category of each scene image and an inspection module corresponding to each inspection category, wherein the inspection module includes at least one inspection object; Acquire location information of the first user at a location corresponding to each of the scene images, at least one guide information, and detection information of each of the inspection objects, where different guide information corresponds to different inspection objects; An inspection execution model of the target site is established based on at least one of the location information, the inspection modules corresponding to each of the location information, the detection information of at least one inspection object in each of the inspection modules, and the guidance information corresponding to each of the inspection objects. The inspection execution model includes an inspection task for at least one of the inspection objects, different inspection objects correspond to different location information and different guidance information, and the inspection task is generated from the detection information of the inspection object.

2. The method according to claim 1, characterized in that After establishing the inspection execution model of the target site according to at least one of the location information, the inspection modules corresponding to each of the location information, the detection information of at least one inspection object in each of the inspection modules, and the guidance information corresponding to each of the inspection objects, the method further includes: When the second user inspects the target site, obtaining location information and guidance information of the second user; Inputting the location information and guidance information of the second user into the inspection execution model, determining a target inspection task, wherein the target inspection task is an inspection task of an inspection object corresponding to the location information and guidance information of the second user in the inspection execution model, and the target inspection task includes target detection information; Acquire interaction information of the second user performing the inspection action indicated by the target detection information on the target inspection object; An inspection result of the target inspection task is determined according to the interaction information and the target detection information, and the inspection result is used to indicate whether the target inspection task is qualified.

3. The method according to claim 2, characterized in that The target detection information includes normal working conditions and treatment opinions for the normal working conditions and abnormal working conditions and treatment opinions for the abnormal working conditions; The determining the inspection result of the target inspection task according to the interaction information and the target detection information includes: In the case where the interaction information is the normal working condition, determining that the inspection result of the target inspection task is qualified, and sending the processing opinion of the normal working condition to the second user; In the case where the interactive information is the abnormal working condition, the inspection result of the target inspection task is determined to be unqualified, and the handling opinion of the abnormal working condition is sent to the second user.

4. The method according to claim 3, characterized in that The above method further includes: In the case where the interaction information is the abnormal operating condition, generating selection confirmation information and sending it to the terminal device of the second user, the selection confirmation information being used to confirm whether the interaction information is correct; Receiving a confirmation operation sent by the terminal device of the second user; If the confirmation operation indicates that the interaction information is confirmed incorrectly, generating prompt information, the prompt information is used to prompt the second user to perform the target inspection task; In a case where the confirmation operation indicates that the interaction information is confirmed to be correct, the target detection information of the target inspection task in the inspection execution model is adjusted according to the interaction information.

5. The method according to claim 1, characterized in that The step of determining the inspection category of each scene image and the inspection module corresponding to each inspection category according to the scene image set includes: Based on a plurality of inspection categories in a preset initial model, identifying the inspection category of each of the scene images in the scene image set, wherein the initial model further includes inspection modules of the plurality of inspection categories; Among the inspection modules of the plurality of inspection categories, the inspection modules corresponding to the inspection categories of the respective scene images are matched.

6. The method according to claim 1, characterized in that The guidance information includes sight direction, head direction, finger direction and body direction. The inspection execution model of the target site is established according to at least one of the position information, the inspection module corresponding to each of the position information, the detection information of at least one inspection object in each of the inspection modules and the guidance information corresponding to each of the inspection objects, including: An inspection execution model of the target site is established based on at least one of the position information, the inspection modules corresponding to each of the position information, the detection information of at least one inspection object in each of the inspection modules, and at least one of the line of sight direction, the head direction, the finger direction and the body direction in the guidance information corresponding to each of the inspection objects.

7. A device for establishing an inspection task, characterized in that: The device comprises: A first acquisition module, used to acquire a scene image set of a target site, wherein the scene image set includes at least one scene image; A first determination module, configured to determine, according to the scene image set, an inspection category of each scene image and an inspection module corresponding to each inspection category, wherein the inspection module includes at least one inspection object; A second acquisition module is used to acquire the position information of the first user at the position corresponding to each of the scene images, at least one guide information and the detection information of each of the inspection objects, where different guide information corresponds to different inspection objects; A module is established, which is used to establish an inspection execution model of the target site based on at least one of the location information, the inspection modules corresponding to each of the location information, the detection information of at least one inspection object in each of the inspection modules, and the guidance information corresponding to each of the inspection objects, wherein the inspection execution model includes an inspection task of at least one of the inspection objects, different inspection objects correspond to different location information and different guidance information, and the inspection task is generated by the detection information of the inspection object.

8. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for establishing the inspection task as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for establishing an inspection task according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for establishing an inspection task as described in any one of claims 1 to 6.

Images