Man-machine interaction method based on mixed reality technology and related equipment

Abstract

The invention provides a human-computer interaction method based on a mixed reality technology and related equipment, and relates to the technical field of computers. The method comprises the following steps: obtaining target image data, and determining a mixed reality equipment model matched with the target image data; task information is obtained, single model coding and retrieval are carried out on the current task information, and target document information is determined; and outputting a marked part model of the equipment model and corresponding document information according to the equipment model and the target document information. According to the method, only one model is used as the coding model, and a large number of target detection models do not need to be inserted, so that the problem of high data acquisition cost caused by the fact that man-machine interaction of a mixed reality technology needs to acquire all data in an application scene is solved.

Description

Background Art

[0002] In the traditional industrial field, tasks such as equipment maintenance, parts replacement and factory inspection are an indispensable part of business operations. These tasks often require the executors to have deep professional knowledge and rich practical experience to ensure the accuracy, safety and efficiency of the operation. However, with technological innovation and the expansion of enterprise scale, the demand for rapid training of new employees and new scenarios is increasing. The traditional "master-apprentice" training model lacks standardization and replicability, making it difficult to ensure the quality and consistency of training.

[0003] In related technologies, augmented reality (AR) technology has been used to achieve accurate identification of objects in specific scenarios and provide task guidance, greatly improving operation and maintenance efficiency and training effectiveness. Among them, target recognition of AR technology relies heavily on high-cost supervised deep learning, which requires obtaining all data in the application scenario. However, in many actual scenarios, there is a problem that the cost of data acquisition is too high or even impossible to obtain.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a human-computer interaction method and related equipment based on mixed reality technology, which at least to a certain extent overcomes the problem of high data acquisition cost caused by the need to obtain all data in the application scenario in the related technology.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a human-computer interaction method based on mixed reality technology is provided, including: acquiring target image data, determining a mixed reality device model that matches the target image data; acquiring task information, performing single model encoding and retrieval on the current task information, and determining target text information; and outputting a marked component model and corresponding text information of the device model based on the device model and the target text information.

[0008] In some embodiments, before acquiring image data and determining a mixed reality device model that matches the image data, the method further includes: acquiring text information and storing the text information in a vector form in a document vector library; acquiring standard picture information and determining a device picture vector library based on preprocessing the standard picture information; and pre-establishing a three-dimensional model based on the equipment in the scene.

[0009] In some embodiments, the acquiring target image data and determining a mixed reality device model that matches the target image data includes: acquiring the target image data and determining a feature map of the target image data; inputting the feature map into the image vector library for vector similarity comparison to determine the mixed reality device model.

[0010] In some embodiments, inputting the feature map into the image vector library for vector similarity comparison to determine the device model of the mixed reality includes: inputting the feature map into the image vector library for query to determine a first device name; when the first device name is correctly identified, determining the device model of the mixed reality in the pre-established three-dimensional model according to the first device name.

[0011] In some embodiments, the obtaining of task information, single model encoding and retrieval of current task information, and determination of target copy information include: obtaining task information, single model encoding of the task information, and outputting a corresponding embedding vector; performing similarity retrieval on the embedding vector in the document vector library to determine the target copy information.

[0012] In some embodiments, performing a similarity search on the embedded vector in the document vector library to determine the target copy information includes: performing a similarity search on the embedded vector in the document vector library to query multiple document contents whose similarity with the task information meets a preset threshold; performing a first scoring on the multiple document contents according to a preset scoring mechanism to obtain corresponding target scores, and sorting the target scores to obtain a sorting result; determining the first copy information according to the sorting result; inputting the first copy into a pre-established template to output an operation and maintenance inspection process; performing a second scoring on the operation and maintenance inspection process, the second scoring being used to determine whether the operation and maintenance inspection process can solve the problem of the task information; if so, determining the operation and maintenance inspection process as the target copy information.

[0013] In some embodiments, after performing a second score on the operation and maintenance process, wherein the second score is used to determine whether the operation and maintenance process can solve the problem of the task information, the method further includes: if not, re-entering the first document into a pre-established template for a second score; when the number of times the first document is re-entered into the pre-established template for a second score meets a preset threshold, re-performing a first score on the multiple document contents according to a preset scoring mechanism.

[0014] According to another aspect of the present disclosure, a human-computer interaction device based on mixed reality technology is also provided, including: a device model determination module, used to obtain target image data and determine a mixed reality device model that matches the target image data; a target text information determination module, used to obtain task information, perform single model encoding and retrieval on the current task information, and determine the target text information; a text information determination module, used to output the marking component model and corresponding text information of the device model according to the device model and the target text information.

[0015] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned human-computer interaction methods based on mixed reality technology by executing the executable instructions.

[0016] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the human-computer interaction method based on mixed reality technology described above is implemented.

[0017] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned human-computer interaction methods based on mixed reality technology.

[0018] The human-computer interaction method based on mixed reality technology provided in the embodiments of the present disclosure obtains target image data to determine the mixed reality device model that matches the target image data; obtains task information, performs single model encoding and retrieval on the current task information, and determines the target copy information; outputs the marking component model of the device model and the corresponding copy information according to the device model and the target copy information. This application solves the problem of high data acquisition cost caused by the need to obtain all data in the application scenario for human-computer interaction of mixed reality technology by using only one model as the encoding model without inserting a large number of target detection models.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0021] Figure 1 A schematic diagram showing a structure of a human-computer interaction system based on mixed reality technology in an embodiment of the present disclosure;

[0022] Figure 2 A flow chart of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure is shown;

[0023] Figure 3 A flowchart showing a specific example of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure;

[0024] Figure 4 A flowchart showing another specific example of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure;

[0025] Figure 5 A flow chart showing a method for identifying devices and parts using a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure is shown;

[0026] Figure 6 A flowchart showing another specific example of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure;

[0027] Figure 7 A flowchart showing another specific example of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure;

[0028] Figure 8 A flowchart showing another specific example of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure;

[0029] Fig. 9 A flow chart of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure is shown;

[0030] Fig.10 A flow chart showing a method for determining operation and maintenance and overhaul processes using a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure is shown;

[0031] Fig.11 An example diagram showing content that can be seen in a mixed reality according to an embodiment of the present disclosure is shown;

[0032] Fig.12 An example diagram showing the operation and maintenance precautions of a related component in an embodiment of the present disclosure is shown;

[0033] Fig.13 A flowchart showing an example of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure is shown;

[0034] Fig.14A schematic diagram of a human-computer interaction device based on mixed reality technology in an embodiment of the present disclosure is shown;

[0035] Fig.15 A structural block diagram of a computer device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0037] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0038] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0039] DETR: Detection Transformer, an object detection framework (model) based on the Transformer architecture;

[0040] YOLO: You Only Look Once, a real-time target detection algorithm based on deep learning;

[0041] AIGC: Artificial Intelligence Generated Content, AIGC, artificial intelligence generated content;

[0042] Fine-tuning: model fine-tuning;

[0043] ViT: Vision Transformer, a deep learning model based on the Transformer architecture;

[0044] SAM: Segment Anything Model, a powerful image segmentation pre-training model open sourced by Meta AI;

[0045] Saas:Software as a service, software as a service;

[0046] Numpy: Numerical Python, an open source numerical computing extension library for Python language;

[0047] Embedding: Embedding is the professional name for text encoding and vector;

[0048] 1-shot: Single-sample learning refers to the ability of machine learning to recognize an object with only one positive sample.

[0049] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0050] Figure 1 FIG. 1 shows an exemplary application system architecture diagram to which the human-computer interaction method based on mixed reality technology in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0051] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0052] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0053] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0054] In one example of the present disclosure, the terminal device is used to obtain target image data, determine a mixed reality device model that matches the target image data; obtain task information, perform single model encoding and retrieval on the current task information, and determine target text information; and output the marked component model and corresponding text information of the device model based on the device model and the target text information.

[0055] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0056] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0057] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0058] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0059] Figure 2 A flow chart of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure includes the following steps:

[0060] S202, acquiring target image data, and determining a mixed reality device model that matches the target image data.

[0061] It should be noted that the target image data may be image data that needs to be focused on, recognized or processed in a specific task or application, for example, a device image acquired by a mixed reality device.

[0062] In a specific example, the wearable device obtains a feature map of the image seen by the wearable device by inputting processed real-time image data. The feature map will be transmitted to the image vector library for vector similarity comparison, and the device with the highest similarity to the current image (equivalent to the above-mentioned target image data) will be queried (equivalent to the above-mentioned mixed reality device model that matches the target image data).

[0063] S204, obtaining task information, performing single model encoding and retrieval on the current task information, and determining target copy information.

[0064] It should be noted that the above-mentioned task information may be relevant information that a user needs to obtain in order to complete a certain task, such as scenario task information, specifically, a series of tasks and related information that need to be completed in a specific scenario. The above-mentioned single model may be that only one model is used, for example, ViT, SAM. The above-mentioned encoding may be the conversion of text information into a machine-recognizable numerical or vector form, for example, Embedding. The above-mentioned retrieval may be a similarity retrieval. The above-mentioned target copy information may be copy (answer) information for the task information.

[0065] S206: Outputting the marked component model of the equipment model and the corresponding text information according to the equipment model and the target text information.

[0066] It should be noted that the above-mentioned marked component model can be a price tag that enables the component model to be identified, for example, the component will be highlighted and the device will be transparent.

[0067] This application solves the problem of high data acquisition cost caused by the need to obtain all data in the application scenario for human-computer interaction in mixed reality technology by using only one model as the encoding model without inserting a large number of target detection models.

[0068] In one embodiment of the present disclosure, Figure 3 As shown, the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure can determine the three-dimensional model through the following steps before acquiring image data and determining the mixed reality device model matching the image data. Through the semi-supervised recognition method, it is not necessary to label and train all the parts, which can reduce the labeling cost, improve the labeling efficiency, and reduce the dependence on the labeling quality:

[0069] S302, obtaining text information, and storing the text information in a vector form in a document vector library;

[0070] S304, obtaining standard image information, and determining a device image vector library according to preprocessing the standard image information;

[0071] S306, pre-establishing a three-dimensional model according to the devices in the scene.

[0072] It should be noted that the above-mentioned standard image information may be a single sample learning image and its name.

[0073] In one example of the present disclosure, since the single-sample learning method may not be able to fully capture the complexity and diversity of the data set, optimizing the labeled data of specific parts by fine-tuning the encoding model can more effectively improve the performance of the model on these key parts.

[0074] In one example of the present disclosure, before acquiring image data and determining a mixed reality device model that matches the image data, system preparation is performed, wherein the system preparation includes image data preprocessing, user document encoding, image coding model and feature vector library construction, and device 3D modeling.

[0075] Image data preprocessing includes: data acquisition, image data is acquired in real time by mobile terminals or wearable devices; preprocessing methods, image deformation: adjusting the image to the input size required by the encoding model to ensure that the model can correctly process the image data, image normalization: normalizing the pixel values ​​of the image, format conversion: converting the image data into the format required by the encoding model; preprocessing process: the real-time incoming video stream image data is sent to the encoding model for further processing after the above preprocessing methods.

[0076] User document encoding includes: document acquisition, professional documents provided by users, such as operation and maintenance manuals, are an important part of system preparation. Documents usually contain detailed information about the equipment, descriptions of parts and components, and maintenance guides, etc.; document encoding, using a custom language encoding model to encode document content, converting text information in the document into a vector representation to facilitate subsequent retrieval and matching; document storage, storing the encoded document content in full in the document vector library.

[0077] The image coding model and feature vector library include: model selection, selecting the feature processing part of a model as the coding model, such as the Encoder in ViT, which can extract key features in the image for subsequent component recognition and matching; single-sample learning component image processing, the component image taken by the user through single-sample learning is processed by the coding model to extract the feature vector of the component; feature vector storage, the extracted feature vector is saved in the image vector library in the form of "Part Name: [Feature Vector]".

[0078] Equipment 3D modeling includes: equipment identification, in actual scenarios, the system needs to first identify the equipment, which is achieved through image recognition technology or sensor data; 3D modeling, 3D modeling of the identified equipment, the model needs to accurately reflect the shape, size and component location of the equipment; component positioning, in the 3D model, according to the location information of the component, it can be positioned accurately in subsequent operations.

[0079] In a specific example, the process of building a document vector library includes:

[0080] Document upload: Users upload documents to a specified folder. Supported file formats include common document formats such as pdf, txt, and word. At the same time, document format verification is added to ensure that the uploaded document complies with the specified format.

[0081] Document extraction and segmentation: Extract documents from the specified folder through the vector database script. Segment according to the actual document situation (such as chapters or paragraphs). Limited by the number of characters or tokens (the basic unit of source code) allowed to be input by the language model, if the chapter is long, determine the segmentation size in the form of paragraphs or small chapters, and segment the document. At the same time, natural language processing technology (such as paragraph division, sentence segmentation, etc.) can be introduced to segment documents more intelligently. The segmented documents are vectorized and stored through the above-mentioned vector database. Select appropriate vector representation algorithms and vector databases to improve retrieval efficiency and accuracy.

[0082] In a specific example, the process of building a device image vector library includes:

[0083] Prepare standard images and device names: Prepare a standard image and device name for the device to be identified. In addition, you can create a corresponding relationship table between the image and the device name to facilitate subsequent management and maintenance.

[0084] Feature extraction model selection: Based on technical experience or business needs, select the feature extraction model (encoding model) of the image.

[0085] Image preprocessing: preprocess standard images according to the model and business requirements. Preprocessing includes scaling, cropping, normalization, etc.

[0086] Feature extraction and storage: The preprocessed image is input into the encoding model, and the encoding model extracts the feature map of the corresponding image. The feature map and device name are stored in the image vector database in the key-value format of {'device name':[feature map]}.

[0087] Data storage method: According to the actual business situation, the "database" can be in various forms. For example, if the total number of devices is less than 10,000, the data can be directly stored locally in the form of Numpy data. If the number of devices is very large, the data can be stored in a Mysql database. Since the number of devices in actual business scenarios is generally not too large, the present invention preferentially uses Numpy to store the data locally in the .npy binary data format. This solution takes up less memory and is easier to use and migrate.

[0088] In one example of the present disclosure, the above-mentioned three-dimensional modeling can be done using any modeling software or platform as long as the model format can be directly called by the mixed reality device environment.

[0089] In one embodiment of the present disclosure, Figure 4 As shown, the method for obtaining target image data in the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure and determining a mixed reality device model matching the target image data include the following steps, which can significantly improve the real-time, accuracy, robustness and personalization of the system in the mixed reality technology, and bring users a better quality and smoother mixed reality experience:

[0090] S402, acquiring target image data and determining a feature map of the target image data;

[0091] S404: Input the feature map into the image vector library for vector similarity comparison to determine the device model of the mixed reality.

[0092] For example, real-time image data from a wearable device (such as glasses) is received as target image data, and a feature map is obtained after feature extraction of the target image data. The feature map is transferred to the image vector library for vector similarity comparison, and the device with the highest similarity to the current feature map is queried in the image vector library, and the name of the device is obtained. According to the queried device name, the 3D modeling of the device is found in the 3D model library, and the found 3D model is automatically displayed on one side of the wearable device (such as glasses) as a guide for the user. This application does not preset the encoding model. The user only needs to embed the model according to the input and output parameter formats required by the system process, which is plug-and-play.

[0093] In a specific example of the present disclosure, device and component identification (identification of a mixed reality device model that matches target image data) includes:

[0094] When recognizing a real-time image, the image will be placed in the same encoding model (with the same parameters) used to build the image vector library for feature map extraction; the feature map will be compared with the existing image vector database for similarity, and the device name with the closest similarity will be queried.

[0095] If the device name does not match the current device, you need to change the device perspective and use real-time recognition from other angles to determine whether the current device is correct. If it cannot be recognized, an error message is submitted. At the same time, a complete error handling mechanism is established to record and classify unrecognizable device images for subsequent analysis and improvement.

[0096] There are many similarity comparison methods, such as cosine similarity, Manhattan distance, Euclidean distance, etc. Cosine similarity is used to compare document similarity, and the latter two are usually used to group data. Therefore, the present invention gives priority to using cosine similarity and its related variations (similarity comparison tools often have some derived optimization methods). The following formula (1) is an example of cosine similarity calculation:

[0097]

[0098] Among them, A and B are two vectors whose similarity needs to be calculated. i 、b i Then it is the value in vector A and vector B.

[0099] When the device is correctly identified, the 3D model of the device is searched in the built 3D model library according to the device name. According to the characteristics of mixed reality, the 3D model will be intuitively displayed on the viewing side.

[0100] Since the 3D equipment model contains the models of various components, the component models related to the process will be automatically highlighted (this highlighting can be developed directly through Unity and is only related to the front-end display).

[0101] In a specific example, Figure 5 A flow chart of a human-computer interaction method for identifying devices and parts based on mixed reality technology in an embodiment of the present disclosure is shown. Figure 5 As shown, the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure includes the following steps:

[0102] S501, when the real-time image is recognized, a target image is acquired;

[0103] S502, the target image is input into the encoding model Encoder to extract the feature map and determine the feature map;

[0104] S503, performing a similarity comparison between the feature graph and an existing graph vector database;

[0105] S504, querying the device name that is most similar to the feature graph;

[0106] S505, determine whether the queried device name is the same as the device name of the current device, if so, execute S506, if not, execute S507;

[0107] S506, searching for a 3D model of the device in the constructed 3D model library according to the device name;

[0108] S507, changing the device viewing angle, judging whether the current device is correct through real-time recognition from other angles, and submitting an error message if recognition is unsuccessful;

[0109] S508, searching the device name in the 3D model library, and determining whether a 3D model matching the device name can be found in the 3D model library, if a 3D model matching the device name is found, executing S509, if no 3D model matching the device name is found, executing S507;

[0110] S509, determining the 3D model.

[0111] In one embodiment of the present disclosure, Figure 6 As shown, in the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure, the feature map is input into the picture vector library for vector similarity comparison, and the device model of the mixed reality is determined, which can improve the recognition accuracy and reduce the development cost:

[0112] S602, inputting the feature map into the image vector library for query to determine the first device name;

[0113] S604: When the first device name is recognized correctly, a mixed reality device model is determined in a pre-established three-dimensional model according to the first device name.

[0114] For example, the feature graph is compared with the existing image vector database (equivalent to the above-mentioned image vector database) for similarity, and the device name with the closest similarity is queried. If the device name does not match the current device, it is necessary to change the device perspective and determine whether the current device is correct through real-time recognition from other angles. If it cannot be recognized, an error message is submitted.

[0115] In one embodiment of the present disclosure, Figure 7 As shown, the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure obtains task information, performs single model encoding and retrieval on the current task information, and determines the target copy information, which can improve the accuracy and efficiency of information retrieval and support the accumulation and reuse of knowledge:

[0116] S702, obtaining task information, performing single model encoding on the task information, and outputting a corresponding embedding vector;

[0117] S704: perform similarity search on the embedded vector in the document vector library to determine the target copy information.

[0118] In one embodiment of the present disclosure, Figure 8 As shown, in the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure, the embedded vector is searched for similarity in the document vector library, and the target copy information is determined, including the following steps. The scoring mechanism can improve the semantic and logical relevance evaluation, optimize the result sorting and improve user satisfaction:

[0119] S802, performing similarity search on the embedded vector in the document vector library to find out multiple document contents whose similarity with the task information meets a preset threshold;

[0120] S804, performing a first scoring on the contents of the plurality of documents according to a preset scoring mechanism to obtain corresponding target scores, and sorting the target scores to obtain a sorting result;

[0121] S806, determining the first copywriting information according to the sorting result;

[0122] S808, inputting the first document into a pre-established template and outputting an operation and maintenance process;

[0123] S810, performing a second scoring on the operation and maintenance process, where the second scoring is used to determine whether the operation and maintenance process can solve the problem of the task information;

[0124] S812, if (when the second scoring score is greater than or equal to the preset threshold, it is determined that the operation and maintenance process can solve the problem), the operation and maintenance process is determined as the target copy information.

[0125] In one embodiment of the present disclosure, Fig. 9 As shown, in the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure, after performing a second scoring on the operation and maintenance process, the second scoring is used to determine whether the operation and maintenance process can solve the problem of task information, the following steps are also included:

[0126] S902, if not (when the second scoring score is less than the preset threshold, it is determined that the operation and maintenance process cannot solve the problem), the first document is re-entered into the pre-established template to perform the second scoring again;

[0127] S904: When the number of times the first document is re-entered into the pre-established template for the second scoring meets a preset threshold, the first scoring is performed again on the contents of the plurality of documents according to the preset scoring mechanism.

[0128] In a specific example, agent-based process generation includes:

[0129] Task description encoding: Receive a natural language task description, such as “Please tell me how to perform regular maintenance on the first device?” and convert the task description into an Embedding vector using an encoding method consistent with the text vector library (such as GPT4AllEmbeddings).

[0130] Similarity comparison and document query: The converted task description vector is compared with the established document vector database to query the K documents that are most relevant to the task description. The similarity calculation method can use cosine similarity, Euclidean distance and other methods.

[0131] First scoring: Score and evaluate the most relevant copy content found in the query. The first scoring mechanism is based on the semantic and logical evaluation of the language large model to ensure that the query content is highly relevant to the question asked. If the relevance is not high, score multiple pieces of content one by one to find the most relevant content. The fourth step is template generation: the most relevant document content or paragraph text is passed as input to the Agent, and the Agent generates task-related operation and maintenance and repair processes based on the pre-written Prompt template. Among them, the Prompt template format is divided into three parts: indicating that the Agent's role is a specialist in generating task solution processes; instructing the Agent to generate a process that can solve the task problem raised based on the original content found in the query; and restricting the Agent to answer in a process-based manner to avoid too much text description. The Agent generates a process description based on the most relevant copy content found in the first scoring query.

[0132] Second scoring: Score and evaluate the generated process to determine whether the generated process and the content in the original document can really solve the problem encountered in the task. If the problem cannot be solved, regenerate the process. If a valid process cannot be generated after a preset number of attempts, return to the content query and reselect relevant content.

[0133] This application uses the second scoring (the Agent scores the generated process twice) to more accurately evaluate whether the process truly provides an effective solution to the problem in the task, ensuring the pertinence and effectiveness of the process. If the Agent determines that the generated process cannot solve the problem, the system can regenerate the process. This process is actually an iterative optimization process. Through continuous attempts and adjustments, more complete and effective solutions are gradually generated, which helps to improve the quality of the final output process.

[0134] In a specific example, Fig.10 A flow chart of determining the operation and maintenance process by a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure is shown as follows: Fig.10 As shown, the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure includes the following steps:

[0135] S1001, coding the task instructions;

[0136] S1002, performing a similarity comparison on the converted task description vector in the established document vector database;

[0137] S1003, querying the K documents that are most relevant to the task description;

[0138] S1004, query scoring, the most relevant copy content found out will enter the scoring mechanism, if the query scoring fails, execute S1003, if the query scoring passes, execute S1005;

[0139] S1005, determining the most relevant content based on the scores of the K documents;

[0140] S1006, after the most relevant document content or paragraph text is found, the query result is transmitted to the Agent, and the Agent process is generated;

[0141] S1007, scoring of the Agent process. If the Agent process score does not meet the preset score, S1003 is executed. If the Agent process score meets the preset score, S1008 is executed.

[0142] S1008, generate operation and maintenance and repair process.

[0143] In one example of the present disclosure, based on the device model and the target text information, outputting the marked component model and corresponding text information of the device model includes: highlighting the device model, and the operation and maintenance personnel perform relevant operations in reality in combination with the highlighted component model and the target text information. For example, based on the device model and the target text information, the marked component model of the device model is output. The operation and maintenance personnel can see the highlighted component model through mixed reality devices (such as AR glasses, tablet computers, etc.), and the highlighting helps the operation and maintenance personnel quickly locate the components that need to be operated or inspected.

[0144] Specifically, based on the above output, the following two pieces of information were obtained: 1. The current equipment and parts model (equivalent to the above equipment model); 2. The operation and maintenance process (equivalent to the above document information), where each process contains one or more subtasks. Then, you only need to highlight the relevant parts mentioned in the subtask, and the operation and maintenance personnel will perform the relevant operations in reality based on the highlighted parts model and subtask document description. The document information describes in detail the operation steps and precautions for each subtask to ensure that the operation and maintenance personnel can perform them correctly. Fig.11 This is an example of what can be seen in mixed reality. Based on the highlighted component models (controllers, connectors) in mixed reality, maintenance personnel can find the parts in reality according to the corresponding locations. As with device recognition, when a part is picked up, target recognition will automatically identify whether the picked up part is the one mentioned. If the maintenance personnel pick it up correctly, the maintenance precautions for the relevant parts will be displayed (for example, pay attention to the fact that the pins and crimping edges of the male and female connectors should not be subjected to force). Fig.12(Example diagram of maintenance precautions for related parts) is shown; if the maintenance personnel take the wrong part, an error prompt will be given to guide the maintenance personnel to reselect the correct part. This application adds multi-stage scoring, evaluation and backtracking capabilities to improve the quality of the final generated process answer.

[0145] In a specific example, Fig.13 A flowchart showing an example of a human-computer interaction method based on mixed reality technology in an embodiment of the present disclosure is shown as follows: Fig.13 As shown, the human-computer interaction method based on mixed reality technology provided in the embodiment of the present disclosure includes the following steps:

[0146] S1301, obtaining MR images for preprocessing;

[0147] S1302, input Encoder to determine feature map;

[0148] S1303, inputting the feature map into the image vector library to perform similar feature recognition;

[0149] S1304, determining the device (name) with the highest current image similarity;

[0150] S1305, according to the queried device name, find the 3D model of the device in the 3D model library;

[0151] S1306, based on the current scene task (equivalent to the above task information), encode the scene, task and task description to generate a corresponding code;

[0152] S1307, the encoding of the task description will be searched for similarity in the document vector database, the content closest to the question will be retrieved in the professional document vector database, and the content will be fed back to the Agent;

[0153] S1308, Agent performs task decomposition;

[0154] S1309, scoring and sorting the disassembled tasks;

[0155] S1310, process parts highlighting, obtain the output results of S1305 and S1309, match the parts mentioned in the current subtask process in the task process in the 3D model, the matched parts in the 3D model will be highlighted, and the equipment will be made transparent to intuitively display the relevant parts. The operation and maintenance personnel can intuitively see the parts related to the process and realize the visualization of process guidance.

[0156] It should be noted that an asynchronous mechanism is added for real-time target detection and task process inquiry, that is, the above S1301-S1305 and S1306-S1309 are executed asynchronously. Since real-time target detection requires continuous recognition of components in the current input frame, and task disassembly only requires requesting the Agent once. For example, when the task process is generated for the first time and the first task is executed, the component recognition task will be started and continuously executed. When the user picks up the bright component according to the guidance in the 3D model, the recognition task automatically recognizes that a certain component exists in the current interface, and displays the precautions for the component based on the description of the component in the document. When a task is completed, the user completes the next task by clicking on the task.

[0157] This application is based on the Agent framework of the big model. It uses knowledge acquisition to query the operation and maintenance manual content related to the operation and maintenance scenario, converts the manual content into the operation and maintenance process through the language big model, and describes the subtasks of each process in detail. Finally, according to the generated subtasks, it realizes the capabilities of automatic real-time calculation, drawing instructions and instructions based on the current scene and image. This application no longer requires tedious pre-process customization and text description writing.

[0158] Based on the same inventive concept, the embodiments of the present disclosure also provide a human-computer interaction device based on mixed reality technology, as described in the following embodiments. Since the principle of solving the problem in the embodiment of the device is similar to that in the above method embodiment, the implementation of the embodiment of the device can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0159] Fig.14 A schematic diagram of a human-computer interaction device based on mixed reality technology in an embodiment of the present disclosure is shown. Fig.14 As shown, the apparatus includes: a device model determination module 141, a target text information determination module 142 and a text information determination module 143.

[0160] The device model determination module 141 is used to obtain target image data and determine a mixed reality device model that matches the target image data;

[0161] The target copy information determination module 142 is used to obtain task information, perform single model encoding and retrieval on the current task information, and determine the target copy information;

[0162] The document information determination module 143 is used to output the marked component model of the equipment model and the corresponding document information according to the equipment model and the target document information.

[0163] In one example of the present disclosure, the human-computer interaction device based on mixed reality technology also includes a three-dimensional model building module, which is used to obtain text information and store the text information in a vector form in a document vector library; obtain standard image information, and determine a device image vector library based on preprocessing the standard image information; and pre-build a three-dimensional model based on the equipment in the scene.

[0164] In one example of the present disclosure, the device model determination module is further used to obtain target image data and determine a feature map of the target image data; input the feature map into a picture vector library for vector similarity comparison to determine a device model for mixed reality.

[0165] In one example of the present disclosure, the device model determination module is also used to input the feature map into the image vector library for query to determine the first device name; when the first device name is recognized correctly, the mixed reality device model is determined in a pre-established three-dimensional model according to the first device name.

[0166] In one example of the present disclosure, the target text information determination module is further used to obtain task information, perform single model encoding on the task information, and output a corresponding embedding vector; perform similarity retrieval on the embedding vector in a document vector library to determine the target text information.

[0167] In one example of the present disclosure, the target copy information determination module is further used to perform a similarity search on the embedded vector in the document vector library, and query multiple document contents whose similarity with the task information meets a preset threshold; perform a first scoring on the multiple document contents according to a preset scoring mechanism to obtain a corresponding target score, and sort the target scores to obtain a sorting result; determine the first copy information based on the sorting result; input the first copy into a pre-established template, and output an operation and maintenance inspection process; perform a second scoring on the operation and maintenance inspection process, and the second score is used to determine whether the operation and maintenance inspection process can solve the problem of the task information; if so, determine the operation and maintenance inspection process as the target copy information.

[0168] In one example of the present disclosure, the above-mentioned target copy information determination module is also used to, when the operation and maintenance process cannot solve the problem of task information, re-enter the first copy into a pre-established template for a second score; when the number of times the first copy is re-entered into the pre-established template for a second score meets a preset threshold, re-perform the first score on multiple document contents according to the preset scoring mechanism.

[0169] It should be noted that the above-mentioned device model determination module 141, target text information determination module 142 and text information determination module 143 correspond to S202 to S206 in the method embodiment, and the above-mentioned modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0170] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0171] Refer to the following Fig.15 15 to describe the electronic device 1500 according to this embodiment of the present disclosure. Fig.15 The electronic device 1500 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0172] like Fig.15 As shown, the electronic device 1500 is in the form of a general computing device. The components of the electronic device 1500 may include but are not limited to: at least one processing unit 1510, at least one storage unit 1520, and a bus 1530 connecting different system components (including the storage unit 1520 and the processing unit 1510).

[0173] The storage unit stores program codes, which can be executed by the processing unit 1510, so that the processing unit 1510 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0174] For example, the processing unit 1510 can execute the following steps of the above-mentioned method embodiment: obtain target image data, determine the mixed reality device model that matches the target image data; obtain task information, perform single model encoding and retrieval on the current task information, and determine the target text information; based on the device model and the target text information, output the marked component model and corresponding text information of the device model.

[0175] The storage unit 1520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 15201 and / or a cache storage unit 15202 , and may further include a read-only storage unit (ROM) 15203 .

[0176] The storage unit 1520 may also include a program / utility 15204 having a set (at least one) of program modules 15205, such program modules 15205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0177] Bus 1530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0178] The electronic device 1500 may also communicate with one or more external devices 1540 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1500, and / or communicate with any device that enables the electronic device 1500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1550. Furthermore, the electronic device 1500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 1560. As shown, the network adapter 1560 communicates with other modules of the electronic device 1500 via a bus 1530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0179] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0180] In particular, according to an embodiment of the present disclosure, the process described in the above reference flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above-mentioned human-computer interaction method based on mixed reality technology when executed by a processor.

[0181] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the above method of the present disclosure is stored thereon. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above “Exemplary Method” section of this specification.

[0182] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0184] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0185] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0186] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0187] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0188] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0189] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A human-computer interaction method based on mixed reality technology, characterized in that: include: Acquire target image data, and determine a mixed reality device model that matches the target image data; Obtain task information, perform single model encoding and retrieval on the current task information, and determine the target copy information; According to the device model and the target text information, a marked component model of the device model and corresponding text information are output.

2. The human-computer interaction method based on mixed reality technology according to claim 1, characterized in that: Before acquiring the image data and determining the mixed reality device model matching the image data, the method further includes: Acquire text information, and store the text information in a document vector library in the form of a vector; Acquire standard image information, and determine a device image vector library based on preprocessing the standard image information; Pre-build a 3D model based on the equipment in the scene.

3. The human-computer interaction method based on mixed reality technology according to claim 2, characterized in that: The acquiring target image data and determining a mixed reality device model matching the target image data comprises: Acquire target image data, and determine a feature map of the target image data; The feature map is input into the image vector library for vector similarity comparison to determine the device model of the mixed reality.

4. The human-computer interaction method based on mixed reality technology according to claim 3, characterized in that: The inputting the feature map into the picture vector library to perform vector similarity comparison and determining the mixed reality device model comprises: Inputting the feature graph into the picture vector library for query to determine the first device name; When the first device name is recognized correctly, the mixed reality device model is determined in the pre-established three-dimensional model according to the first device name.

5. The human-computer interaction method based on mixed reality technology according to claim 2, characterized in that: The obtaining of task information, performing single model encoding and retrieval on the current task information, and determining the target copy information includes: Obtaining task information, performing single model encoding on the task information, and outputting a corresponding embedding vector; The embedding vector is searched for similarity in the document vector library to determine the target copy information.

6. The human-computer interaction method based on mixed reality technology according to claim 5, characterized in that: The performing similarity search on the embedded vector in the document vector library to determine the target copy information includes: Perform similarity retrieval on the embedded vector in the document vector library to find out multiple document contents whose similarity with the task information meets a preset threshold; Performing a first scoring on the contents of the plurality of documents according to a preset scoring mechanism to obtain corresponding target scores, and sorting the target scores to obtain a sorting result; Determining first copywriting information according to the sorting result; Input the first document into a pre-established template to output an operation and maintenance process; Performing a second score on the operation and maintenance process, wherein the second score is used to determine whether the operation and maintenance process can solve the problem of the task information; If so, the operation and maintenance process is determined as the target document information.

7. The human-computer interaction method based on mixed reality technology according to claim 6, characterized in that: After performing a second scoring on the operation and maintenance process, wherein the second scoring is used to determine whether the operation and maintenance process can solve the problem of the task information, the method further includes: If not, the first text is re-entered into the pre-established template for a second scoring; When the number of times the first document is re-input into the pre-established template for the second scoring meets a preset threshold, the first scoring is performed again on the contents of the plurality of documents according to a preset scoring mechanism.

8. A human-computer interaction device based on mixed reality technology, characterized in that: include: A device model determination module, used to obtain target image data and determine a mixed reality device model that matches the target image data; The target copy information determination module is used to obtain task information, perform single model encoding and retrieval on the current task information, and determine the target copy information; The document information determination module is used to output the marked component model of the equipment model and the corresponding document information according to the equipment model and the target document information.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the human-computer interaction method based on mixed reality technology as described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the human-computer interaction method based on mixed reality technology described in any one of claims 1 to 7 is implemented.

Images