Data information processing method, device, intelligent operation tool and display device

Abstract

The embodiment of the application provides a data information processing method and device, an intelligent operation tool and a display device, which can be applied to the technical field of data processing and augmented display. The method comprises the following steps: acquiring a first signal, wherein the first signal is from a tool end sensing device integrated in an operation tool, and the first signal at least comprises local perspective information collected by the sensing device and directed to an operation object; determining, based on the local perspective information, identification information of an operated part in the operation object by using a model; determining operation guide data associated with the identification information; and generating display information, wherein the display information is used for superimposedly presenting the operation guide data on a spatial position corresponding to the operated part through a display interface.

Description

Technical Field

[0001] This application relates to the field of data processing and enhanced display technology, and more specifically to a data information processing method, apparatus, intelligent operating tool, and display device. Background Technology

[0002] With the development of industrial manufacturing and equipment maintenance, the assembly and repair of complex products are becoming increasingly frequent, placing higher demands on the professional skills of operators. Traditional operation instructions rely on paper manuals or pre-recorded videos, which suffer from problems such as unintuitive information presentation and a disconnect from the actual operation. Operators need to repeatedly switch their gaze between the instruction materials and the actual object, resulting in low efficiency and a high risk of errors due to misunderstandings. Summary of the Invention

[0003] In view of the above problems, this application provides a data information processing method, device, intelligent operating tool and display device.

[0004] According to a first aspect of this application, a data information processing method is provided, comprising: acquiring a first signal, the first signal being from a tool-end sensing device integrated into an operating tool, the first signal including at least local perspective information collected by the sensing device and pointing to an operating object; determining, based on the local perspective information, identification information of a part to be operated in the operating object using a model; determining operation guidance data associated with the identification information; and generating display information, the display information being used to overlay the operation guidance data onto a spatial position corresponding to the part to be operated via a display interface.

[0005] According to an embodiment of this application, the first signal further includes interaction information characterizing the physical interaction between the operating tool and the operated part; the method further includes: determining the current stage of the physical interaction based on the interaction information, determining corresponding stage guidance information from the operation guidance data based on the current stage, and updating the stage guidance information to the display information.

[0006] According to an embodiment of this application, the method further includes: acquiring a second signal, the second signal being from a head-mounted sensing device, the second signal including at least a first perspective information characterizing the user's operating environment containing the operating object; the generation of display information includes: generating the display information based on the first perspective information, such that the operation guidance data is embedded in the first perspective information at a position corresponding to the operated part.

[0007] According to an embodiment of this application, the method further includes: processing the first perspective information based on the display information to generate a first image; and sending the first image to a display terminal so that the display terminal displays the first image.

[0008] According to an embodiment of this application, the method further includes generating a second screen based on the local perspective information; generating a multi-view interface, the multi-view interface being configured to simultaneously display the first screen and the second screen; and switching the main display content of the multi-view interface between the first screen and the second screen in response to a perspective switching command triggered on the multi-view interface.

[0009] According to an embodiment of this application, determining the operation guidance data associated with the identification information includes: receiving an operation instruction to perform a target operation on the operated part; determining an operation step sequence using a model based on the operation instruction and the identification information; and determining the operation guidance data based on the operation step sequence, wherein the operation guidance data is at least used to guide the user to complete the target operation.

[0010] A second aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs; wherein the one or more processors execute the one or more computer programs to perform the following operations: acquiring a first signal from a tool-end sensing device integrated into an operating tool, the first signal including at least local view information collected by the sensing device and pointing to an operating object; determining, based on the local view information, identification information of a part to be operated on the operating object using a model; determining operation guidance data associated with the identification information; and generating display information for overlaying the operation guidance data onto a spatial position corresponding to the part to be operated on through a display interface.

[0011] A third aspect of this application provides an intelligent operation tool, comprising: a tool body; a sensing module integrated into the tool body for acquiring local perspective information of the operation object when the tool body points to at least one operation object; and a processing and communication module coupled to the sensing module, configured to: acquire the local perspective information; determine the identification information of the operated part in the operation object using a model based on the local perspective information; send the identification information to a terminal device so that the terminal device determines operation guidance data associated with the identification information; and generate display information, wherein the display information is used to overlay the operation guidance data at a spatial position corresponding to the operated part in a user's first perspective.

[0012] According to an embodiment of this application, the processing and communication module is further configured to: acquire interaction information when the operating tool and the operated part physically interact; determine the current stage of the physical interaction based on the interaction information; determine corresponding stage guidance information from the operation guidance data based on the current stage; and send the stage guidance information and the identification information to the terminal device so that the terminal device updates the display information according to the stage guidance information.

[0013] A fourth aspect of this application provides a head-mounted display device, comprising: a communication module configured to receive target identification information from an intelligent operating tool, the target identification information being determined by the intelligent operating tool using a model based on collected local viewpoint information of the operating object; a processing module coupled to the communication module, for acquiring operation guidance data associated with the target identification information; generating display information, the display information being used to overlay the operation guidance data onto a spatial position in a user's first viewpoint corresponding to the operation part indicated by the target identification information; and a display module for outputting the display information. Attached Figure Description

[0014] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0015] Figure 1 This illustration schematically depicts an application scenario of the data information processing method according to an embodiment of this application.

[0016] Figure 2 A flowchart illustrating a data information processing method according to an embodiment of this application is shown schematically.

[0017] Figure 3 A flowchart illustrating another data information processing method according to an embodiment of this application is shown schematically;

[0018] Figure 4 A flowchart illustrating another data information processing method according to an embodiment of this application is shown schematically;

[0019] Figure 5 A schematic diagram of a multi-view interface according to an embodiment of this application is shown;

[0020] Figure 6 A schematic diagram of the structure of the intelligent operating tool according to an embodiment of this application is shown;

[0021] Figure 7 A schematic diagram of a head-mounted display device according to an embodiment of this application is shown.

[0022] Figure 8A schematic block diagram of an example electronic device that can be used to implement embodiments of this application is shown. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] Figure 1 The diagram illustrates an application scenario of the data information processing method according to an embodiment of this application.

[0028] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a display terminal 110, an operation object 120, an operator 130, a head-mounted sensing device 140, an operation tool 150, a processing device 160, and a remote collaborator 170.

[0029] The object to be operated on 120 can be the target entity that needs to be operated on. For example, the object to be operated on 120 can be complex industrial equipment, electronic products, mechanical devices, medical equipment, vehicle components, or any physical entity that needs to be disassembled, repaired, assembled, or maintained.

[0030] Operator 130 may use handheld tools 150 to perform physical operations on the object 120. For example, operator 130 may be a product development engineer, production line assembly worker, equipment maintenance technician, field service engineer, end user, or any person who needs to perform physical operations. The operating environment of operator 130 may be the actual physical space where the physical operations are performed, such as a product development laboratory, production workshop, equipment maintenance room, field work site, home environment, or any place that requires physical operation guidance.

[0031] The head-mounted sensing device 140 is worn on the head of the operator 130 and is capable of collecting environmental information within the operator's field of vision. For example, the head-mounted sensing device 140 can be augmented reality glasses, virtual reality helmets, smart helmets, or any head-mounted device with image acquisition and information display functions.

[0032] The operating tool 150 is held by the operator 130 and physically interacts with the object 120. The operating tool 150 can collect relevant information during the interaction between the operating tool 150 and the object 120. For example, the operating tool 150 can be an electric screwdriver, wrench, multimeter, welding equipment, testing instrument, or any tool used for physical interaction with the object.

[0033] The processing device 160 can be located within the operating environment or in the cloud, and is communicatively connected to the head-mounted device 140 and the operating tool 150 to receive and process data from the head-mounted sensing device 140 and the operating tool 150. For example, the processing device 160 can be a local computer, an edge server, a cloud server, or any device with data processing capabilities.

[0034] Remote collaborator 170 is located at a remote location different from the operating environment. Through a display terminal 110, they communicate with the processing device 160, enabling them to view relevant information at the operating site in real time and communicate with the operator 130 via voice or visual means. For example, remote collaborator 170 could be a product design expert, maintenance guidance expert, quality auditor, training instructor, or anyone who needs to remotely participate in the operation process.

[0035] Display terminal 110 is a device used by remote collaborator 170. For example, display terminal 110 can be a desktop computer, laptop computer, tablet computer, smartphone, or any terminal device with display and interactive functions.

[0036] The augmented reality display information generation method provided in this application embodiment can also be executed by a head-mounted sensing device. Accordingly, the related devices provided in this application embodiment can also be disposed in a head-mounted sensing device.

[0037] Alternatively, some steps of the augmented reality display information generation method provided in this application embodiment can be executed by a tool-side sensing device, some steps by a processing device, and some steps by a remote terminal. Correspondingly, the related devices provided in this application embodiment can also be distributed across multiple devices.

[0038] It should be understood that Figure 1 The number of head-mounted sensing devices, operating tools, processing devices, and remote terminals shown is merely illustrative. Depending on actual deployment needs, any number of these components can be used. For example, a single processing device can connect to multiple head-mounted sensing devices and operating tools simultaneously, supporting multi-user collaborative operation scenarios; alternatively, multiple processing devices can be deployed in a distributed manner to jointly undertake data processing tasks.

[0039] It should be noted that the technical solution of this application can be applied to various network communication environments, including but not limited to local area networks, wide area networks, the Internet, mobile communication networks, and satellite communication networks. Communication between the head-mounted sensing device, the tool-end sensing device, the processing equipment, and the remote terminal can be via wired or wireless communication, including but not limited to Wi-Fi, Bluetooth, 5G, 4G, ZigBee, LoRa, and NB-IoT communication protocols.

[0040] Figure 2 A flowchart illustrating a data information processing method according to an embodiment of this application is shown.

[0041] like Figure 2 As shown, the data information processing method 200 of this embodiment includes operations S210 to S240.

[0042] In operation S210, the first signal is acquired.

[0043] According to an embodiment of this application, the first signal comes from a tool-end sensing device integrated into the operating tool, and the first signal includes at least local view information collected by the sensing device and pointing to the operating object.

[0044] According to embodiments of this application, the tool-end sensing device can be an image acquisition unit integrated on the operating tool, such as a miniature camera mounted on the handle of an electric screwdriver, a macro lens integrated into the head of a wrench, or a vision sensor located at the front end of welding equipment. The optical axis direction of this sensing device can be substantially consistent with the operating direction of the operating tool, enabling the sensing device to capture local detail images when the operating tool interacts with the object being operated.

[0045] According to embodiments of this application, local viewpoint information can refer to visual information collected by the sensing device when the operating tool is pointed at the operating object, focusing on a local area of ​​the operating object. This local viewpoint information can have the following characteristics: compared with the user's first viewpoint, the local viewpoint information has a higher magnification and finer detail resolution, and can clearly present the tiny structures on the operating object, such as the threads of a screw, the engagement state of a buckle, and the pins of a connector.

[0046] According to embodiments of this application, the operating tool can be any tool used for physical interaction with the object being operated on, such as a screwdriver, wrench, pliers, multimeter, welding equipment, testing instruments, assembly fixtures, etc. When the operating tool physically interacts with the object being operated on, the actuating end of the operating tool (such as a cutting head, jaws, or probe) comes into contact with or moves relative to the operated part of the object.

[0047] During operation S220, based on local perspective information, the model is used to determine the identification information of the operated part in the operation object.

[0048] According to embodiments of this application, the model can be a recognition model based on deep learning technology, such as a convolutional neural network, an object detection network, or a visual transformer. This model, after pre-training, is capable of recognizing specific objects or components from input images. During the model training phase, comprehensive data collection and learning can be performed on all possible operational objects and their constituent components in the operational scenario, constructing a feature library covering various types of components. This training data can originate from product design drawings, 3D models of components, historical maintenance images, or manually annotated operation records, etc.

[0049] Understandably, since the model has pre-learned all possible components, there is no need for on-site learning of the current object or online searching of each component during the recognition process. Once local viewpoint information is acquired, the model can directly perform feature comparison and matching in the pre-built feature library, quickly outputting the recognition result.

[0050] According to the embodiments of this application, when the local recognition result of the model for local viewpoint information does not meet the preset conditions or the model cannot output valid identification information, the local viewpoint information can be uploaded to a cloud server, and the recognition can be assisted by a more comprehensive cloud database or manual review. The recognition result confirmed by the cloud can be used for subsequent updates of the local model, such as as incremental training data to fine-tune or iterate the local model, so that the model has the effect of continuous learning and expanding recognition capabilities.

[0051] According to embodiments of this application, the model can be deployed locally on a processing device, such as a computing device at the operational site, an edge server, or a processing unit integrated into a head-mounted sensing device. In local deployment, the processing device directly loads the model file and performs inference calculations, completing recognition and decision-making tasks without relying on a network connection. The model can also be deployed on a cloud server or server cluster. In cloud deployment, the processing device uploads the collected local viewpoint information to the cloud via a communication network. After the cloud server performs model inference calculations, it returns the recognition results or operational guidance data to the processing device. The model can also be deployed in a hybrid manner, with some models deployed locally and some deployed in the cloud.

[0052] According to embodiments of this application, the operated part can refer to a specific component of the object being operated on, which the operating tool is currently pointing at or about to operate. For example, in the scenario of repairing a laptop computer, the operated part can be a specific component such as a "bottom shell fixing screw", a "memory slot clip", or a "fan power interface".

[0053] According to embodiments of this application, the identification information can be information used to uniquely identify a specific part within the operation object. For example, the identification information may include one or more of the following component-related information: component code, component name, component category, component model, component attributes, and component location information. For example, the component code is the part number in the product bill of materials. For example, the component name may be "CPU heatsink mounting screw" or "motherboard power connector." For example, the component category may be screws, clips, connectors, or cables. For example, the component attributes may include specifications, material, color, and size parameters; the component model may be a numeric or alphanumeric code used to identify the component, such as ISO 4017 (hexagonal head bolts, international standard); and the component location information may be the relative position coordinates or topological relationship of the component within the operation object.

[0054] In operation S230, determine the operation guidance data associated with the identification information.

[0055] According to embodiments of this application, operation guidance data is information content used to assist users in performing operations on the operated parts. For example, operation guidance data may include text descriptions, graphic illustrations, parameter values, and multimedia content. For example, text descriptions may include phrases like "Use a Phillips screwdriver to rotate counterclockwise three times" or "Lift the clip upwards until you hear a click," etc. Graphic illustrations may include component structure diagrams, disassembly / assembly sequence diagrams, exploded views, etc. Parameter values ​​may include target torque values, rotation angles, insertion depths, etc. Multimedia content may include short operation demonstration videos, animations, voice prompts, etc.

[0056] According to an embodiment of this application, the association between operation guidance data and identification information can be pre-established and stored in a database. Once the identification information is obtained, the corresponding operation guidance data can be obtained by querying the database.

[0057] In operation S240, display information is generated. The display information is used to overlay operation guidance data onto the spatial position corresponding to the operated part through a display interface.

[0058] According to embodiments of this application, display information can be a set of instructions or a data stream used to control a display device to present visual content. For example, display information may include content to be presented, such as text, graphics, animations, highlighted areas, and other visual elements. For example, display information may include presentation position information, indicating the position where the visual element should appear in the display interface. For example, display information may include presentation style information, such as color, size, transparency, and animation effects. For example, display information may include presentation timing information, such as display duration, appearance timing, and disappearance conditions.

[0059] According to embodiments of this application, a display interface refers to a physical or logical surface on which a user can view visual information. Examples include the see-through display of a head-mounted display device, the computer screen of a remote terminal, the touchscreen of a mobile device, and the projection surface of a projection device.

[0060] According to embodiments of this application, overlay presentation can refer to the spatial fusion of virtually generated operation guidance data with the real environment or environmental images seen by the user, making the operation guidance data appear to be attached to a specific location of the real operation object. For example, overlay presentation can be achieved through optical perspective, video perspective, remote video stream overlay, etc.

[0061] According to embodiments of this application, the spatial position corresponding to the operated part can refer to a definite correspondence between the presentation position of the operation guidance data and the position of the operated part in real space. For example, this correspondence can be determined through image coordinate mapping, three-dimensional pose coordinate transformation, spatial relationship anchoring, etc.

[0062] According to the embodiments of this application, the process from acquiring local viewpoint information to generating display information can be completed in real-time or near real-time. Specifically, parameters such as the acquisition frequency of local viewpoint information, model recognition speed, and generation and transmission delay of display information can be configured according to the actual application scenario to ensure that users have a smooth operating experience.

[0063] According to the embodiments of this application, on the one hand, since component identification is performed directly based on local perspective information collected by the tool, compared with methods relying on user subjective judgment or remote expert manual identification, the accuracy and efficiency of identifying the operated part can be improved, avoiding the risk of misoperation due to identification errors. On the other hand, by presenting the operation guidance data in a spatial overlay manner at the position corresponding to the operated part, an intuitive spatial correspondence is established between the operation guidance information and the physical operation object. Users do not need to repeatedly shift their gaze between the guidance information and the operation object, effectively reducing cognitive load and operation error rate. Furthermore, the entire identification, association, and presentation process is completed automatically without real-time human intervention, thereby eliminating the dependence on continuous online guidance from remote experts, significantly improving the autonomy and response efficiency of operation assistance, and is particularly suitable for application scenarios with limited network conditions or scarce expert resources.

[0064] Figure 3 A flowchart illustrating another data information processing method according to an embodiment of this application is shown.

[0065] like Figure 3 As shown, in this embodiment, the first signal may further include interaction information characterizing the physical interaction between the operating tool and the operated part. The data processing method 300 of this embodiment includes operations S310 to S330.

[0066] When operating S310, the current stage of the physical interaction is determined based on the interaction information.

[0067] In some embodiments, interaction information may refer to dynamic parameter data generated by the operating tool during physical interaction. Interaction information may include one or more types of mechanical parameters, kinematic parameters, state parameters, and time parameters.

[0068] For example, mechanical parameters can be mechanical data such as force, torque, and pressure applied by the operating tool. Examples include the real-time torque value generated by an electric screwdriver during screw tightening, and the magnitude of the torque applied by a wrench when turning a nut.

[0069] For example, kinematic parameters can be motion data such as changes in position, angle, displacement, velocity, and acceleration of the operating tool. Examples include the rotation angle of a screwdriver, the forward and backward displacement of a socket wrench, and the travel distance of a crimping tool.

[0070] For example, status parameters can be the current working status of the tool, such as working mode (tightening / loosening), gear setting, battery level, etc.

[0071] For example, time parameters can be timing information such as the duration of the interaction process and the operation interval.

[0072] In some embodiments, the interaction information can be acquired by various sensors integrated on the operating tool. Examples include torque sensors, pressure sensors, inertial measurement units, angle sensors, and displacement sensors. These sensors can perceive the physical interaction between the operating tool and the operated part in real time and transmit the acquired data as part of a first signal to a processing device.

[0073] In some embodiments, the current stage of physical interaction can refer to the specific step or state in which the operating tool interacts with the operated part. A complete physical interaction process can be decomposed into multiple consecutive stages, each with a specific operational purpose and state characteristics. For example, taking the tightening of screws in laptop repair as an example, it can be divided into the alignment stage, the initial screwing stage, the tightening stage, the final tightening stage, and the overload warning stage. The core characteristics and operational requirements of each stage are clearly different.

[0074] In some embodiments, determining the current stage based on interaction information can employ threshold judgment, state machine modeling, learning methods, etc. For example, machine learning methods can utilize pre-trained machine learning models, such as Hidden Markov Models or Recurrent Neural Networks, inputting the interaction information sequence into the model, and having the model output the current stage.

[0075] When operating S320, based on the current stage, the corresponding stage guidance information is determined from the operation guidance data.

[0076] In some embodiments, stage guidance information may refer to operational instructions corresponding to a specific stage of physical interaction. Operational instruction data can be organized and stored according to interaction stages, with each stage associated with specific guidance information.

[0077] For example, taking the five stages of tightening a screw as an example, the guidance information for each stage needs to match the core operational requirements: the alignment stage focuses on the alignment guidance of tools and components, the tightening stage focuses on real-time prompts of torque changes, and the overload warning stage focuses on risk warnings and operation stop instructions.

[0078] In some embodiments, the specific way to determine the stage guidance information may be to query the corresponding guidance content from a predefined operation guidance data table according to the current stage, or to dynamically generate guidance content that matches the operation requirements of the current stage, or to select or generate more refined guidance content based on the specific values ​​of the current stage and the interaction information.

[0079] For example, based on the current stage, the corresponding guidance content can be directly queried from a predefined data table. This data table is pre-configured with fixed guidance information for each stage. Current stage: Initial screwing stage (the screw begins to enter the threaded hole). Predefined content: The data table query yields the text prompt "Rotate clockwise at a constant speed, keeping the cutter head perpendicular," and a clockwise rotating arrow animation is displayed on the screen.

[0080] For example, based on the current stage, the system generates guidance content that matches the operational requirements of that stage. Current stage: Tightening stage (screw contacts the bottom surface, torque begins to rise). The generated guidance content is as follows: The system dynamically generates a real-time progress bar with "Target torque value: 0.4 N·m, Current torque: 0.15 N·m" based on the screw specification (M2.5) and material (aluminum alloy), and generates an adaptive prompt "Note that the resistance is gradually increasing".

[0081] For example, based on the specific values ​​of the current stage and interactive information, more refined guidance content can be selected or generated. Current stage: Tightening stage, real-time torque value is 0.35 N·m. Refined processing: When the torque value is within the range of 0.30-0.38 N·m, display "Approaching target value, decelerate rotation"; when the torque value exceeds 0.42 N·m, display a red warning "Too much torque, please stop immediately and check if the thread is damaged"; when the torque increase rate is abnormal, additionally display a correction prompt "Abnormal resistance detected, please check if the screw is tilted".

[0082] When operating S330, update the stage boot information to the displayed information.

[0083] In some embodiments, updating the stage guidance information to the display information may refer to replacing, overlaying, or adjusting the style of the guidance content corresponding to the current stage and integrating it into the original display information, so that the guidance content presented on the display interface can be dynamically updated as the interaction stage changes.

[0084] For example, updating the displayed information can be done by replacing the previous stage's guidance information with the current stage's guidance information, thus maintaining consistency between the displayed content and the current operation stage.

[0085] For example, updating the displayed information can involve overlaying content, adding guidance elements specific to the current stage, such as progress bars, target value indicators, and warning signs, on top of the existing displayed information.

[0086] For example, updating the displayed information could involve style changes. Depending on the stage, the presentation style of the guidance information could be altered, such as color changes, flashing effects, or size changes, to indicate to the user that the operation stage has changed.

[0087] For example, updated display information can disappear and reappear; the guidance information for certain stages may disappear automatically after entering the next stage, while new guidance information appears.

[0088] In one example, during the screw tightening process, the displayed information can dynamically change as the stages progress. For instance, the initial stage displays alignment prompts and screw specification information, the screwing-in stage displays rotation direction and speed indicators, the tightening stage displays a progress bar showing the real-time torque value and the target torque value, and the completion stage displays a "Done" message and instructions for the next step.

[0089] According to the embodiments of this application, on the one hand, since the interactive information can reflect the physical interaction state such as mechanics and kinematics between the operating tool and the operating object in real time, it can accurately determine the current operation stage of the user, thereby implementing staged guidance. Staged guidance can provide the most relevant and timely guidance content for the specific operation task faced by the user, avoiding the problems of information overload or information lag. On the other hand, by dynamically updating the staged guidance information to the displayed information, the guidance content and the operation process are synchronized in real time. The user does not need to manually check the guidance information or wait for the next prompt during the operation, making the operation process smoother and more natural.

[0090] Figure 4 A flowchart illustrating another data information processing method according to an embodiment of this application is shown.

[0091] like Figure 4 As shown, the data information processing method 400 of this embodiment includes operations S410 to S420.

[0092] In operation S410, a second signal is acquired from the head-mounted sensing device.

[0093] In some embodiments, a head-mounted sensing device can refer to a device worn on a user's head that has image acquisition capabilities. For example, a head-mounted sensing device can be augmented reality glasses, mixed reality headsets, smart helmets, head-mounted camera modules, etc. This device is typically fixed to the user's head and can change its acquisition direction as the user's head rotates, thereby always acquiring the image in the user's current gaze direction.

[0094] In some embodiments, the second signal includes at least first-viewpoint information characterizing the user's view of the operating environment containing the objects being operated on. The first-viewpoint information may refer to visual information acquired from a head-mounted sensing device that simulates the user's eye perspective. For example, the acquisition direction of the first-viewpoint information may change with the user's head movement, reflecting the user's current field of vision in real time. For example, the first-viewpoint information may cover the user's binocular field of vision, presenting a panoramic view of the operating environment. For example, the first-viewpoint information may reflect the relative positional relationships and spatial layout of objects in the user's field of vision.

[0095] In some embodiments, the first signal and the second signal can be synchronized with timestamps, and the time identifier of the operation time is carried during transmission to ensure the time consistency of multi-source signals.

[0096] During operation S420, display information is generated based on the first-view information so that operation guidance data is embedded in the first-view information at the position corresponding to the operated part.

[0097] In some embodiments, generating display information based on first-viewpoint information can refer to determining how the operation guidance data is presented in the user's field of vision, using the first-viewpoint information as a spatial reference. In a specific example, the position of the operated part in the first-viewpoint information can be determined, the spatial position of the operated part can be mapped to the display coordinates of the display interface, and display information for controlling the display device to present visual content can be generated based on the content of the operation guidance data and the aforementioned position information.

[0098] In some embodiments, embedding the location corresponding to the operated part in the first-view information can refer to establishing a spatial correspondence between the visual presentation of the operation guidance data and the actual location of the operated part in the user's field of vision. This embedding can be manifested as attachment embedding, directional embedding, substitution embedding, environmental embedding, etc.

[0099] According to the embodiments of this application, the display information generated in the above manner essentially constitutes augmented reality display information. The operation guidance data presented by this display information is visually integrated with the real operating environment seen by the user through the head-mounted sensing device, with virtual content and real objects coexisting in the same field of view. The operation guidance data is not fixed to a certain position on the display screen, but maintains a relatively fixed positional relationship with the operated part in the real space. When the user's head turns or moves, the operation guidance data will follow the operated part and move in the screen, always maintaining the correct spatial position. The display information can also be dynamically updated according to the user's real-time operation status, head movement, and gaze direction, providing the user with immediate visual feedback.

[0100] According to an embodiment of this application, a first viewpoint can be generated by processing display information. The first viewpoint is then sent to a display terminal, causing the terminal to display it. Since the first viewpoint is generated by processing the display information, the content of the viewpoint displayed on the remote terminal is consistent with the augmented reality viewpoint seen by the local user through a head-mounted device. Remote viewers can see the overlay effect of operation guidance data in the real environment, intuitively understanding the operation instructions as if they were present at the scene, thus improving the efficiency and accuracy of remote collaboration.

[0101] First-person perspective information typically exists in the form of a video stream, containing continuous images within the user's field of view. Processing first-person perspective information based on display information can refer to using the display information as the basis for rendering, performing image processing or graphic compositing on the first-person perspective information to generate a composite image that blends virtual guidance content with real-world environmental footage.

[0102] The first frame can refer to a processed sequence of images or video frames that incorporates first-person perspective information and operational guidance data. The first frame can include not only the real-world environment from the first-person perspective information but also the virtual guidance content defined by the operational guidance data, with both maintaining a spatial correspondence.

[0103] It should be understood that the generation frequency of the first frame can be configured according to the needs of the application scenario. In scenarios requiring smooth remote viewing, the first frame can be generated continuously in the form of a video stream, such as 30 frames per second; in scenarios where only key frames are needed, still images can be generated on demand.

[0104] Figure 5 A schematic diagram of a multi-view interface according to an embodiment of this application is shown.

[0105] like Figure 5 As shown, in this embodiment, a second screen 520 can be generated based on local viewpoint information. This leads to the generation of a multi-view interface 500 that simultaneously displays the first screen 510 and the second screen 520. In response to a viewpoint switching command triggered on the multi-view interface 500, the main display content of the multi-view interface 500 can be switched between the first screen 510 and the second screen 520.

[0106] As mentioned earlier, the local perspective information features high magnification and fine details, and can clearly present the microstructure when the operating tool interacts with the operated part.

[0107] The second view 520 refers to a visual image generated based on local perspective information, which displays local details of the manipulated object. The second view 520 focuses on the local area where the operating tool interacts with the operated part, and can present details that are difficult to see in the first perspective information, such as the thread state of a screw, the engagement of a buckle, and the alignment of connector pins.

[0108] A multi-view interface 500 refers to a graphical user interface capable of simultaneously displaying multiple visual images. This interface 500 can be displayed on the screen of a display terminal for use by remote viewers.

[0109] In some embodiments, the multi-view interface 500 has at least two display areas simultaneously, used to present a first screen 510 and a second screen 520 respectively. The two screens can be displayed side-by-side, arranged vertically, in a picture-in-picture layout, or any other layout that can display multiple views simultaneously. The first screen 510 and the second screen 520 can be related in content; the first screen 510 displays the overall operating environment from the user's first-person perspective, and the second screen 520 displays close-up details of the tool during the same operation. The first screen 510 and the second screen 520 can be synchronized in time; that is, the second screen 520 displays the details of the position currently pointed to by the operating tool in the first screen 510, and both reflect the operating state at the same moment.

[0110] The triggering methods for the perspective switching command can include: click or touch, shortcut key operation, voice command, gesture operation, automatic triggering, etc. For example, automatic triggering can automatically switch the main screen according to the operation stage, such as automatically zooming in on a local view when a critical operation is detected. For example, when the sensing device detects that the physical interaction parameters (torque / pressure / rotation angle) between the operating tool and the operated part reach a preset threshold, it is determined to be a critical operation stage, and the system automatically triggers the perspective switching command, switching the second screen 520 to the main display content.

[0111] In some embodiments, component information related to the operated part may also be overlaid on the multi-view interface 500. For example, the displayed component information may be component code, component name, component category, component model, component attributes, and component location information as described above. The component information may be presented on the first screen 510 or the second screen 520 of the multi-view interface 500, or simultaneously on the first screen 510 and the second screen 520.

[0112] In some embodiments, the main display content may refer to the screen that occupies the main display area or attracts the user's main attention in a multi-view interface. Switching the main display content may refer to swapping the roles of the current main screen and the secondary screen. For example, when the first screen 510 is the main screen and the second screen 520 is the secondary screen, after executing the switching command, the second screen 520 becomes the main screen and the first screen 510 becomes the secondary screen.

[0113] According to the embodiments of this application, the first screen displays the overall environment from the user's first-person perspective, while the second screen displays local details from the tool's perspective. The two screens complement each other, enabling remote viewers to simultaneously obtain operational information at both macro and micro levels. This helps remote experts more accurately understand the on-site operational situation and potential problems. By supporting dynamic switching of the main display content, remote viewers can flexibly choose which perspective to focus on based on their current focus. When it is necessary to focus on a particular perspective, switching the main display content quickly adjusts the focus, reducing cognitive switching costs and improving information acquisition efficiency.

[0114] According to embodiments of this application, an operation instruction to perform a target operation on the operated part can be received. Based on the operation instruction and identification information, a sequence of operation steps is determined using a model. Based on the sequence of operation steps, at least operation guidance data for guiding the user to complete the target operation is determined. By receiving the user's operation intention, the complete sequence of steps required to complete the target operation can be automatically planned, transforming static component information into dynamic operation guidance, achieving an intelligent closed loop from recognition to guidance. The user does not need to know the specific steps of a complex operation in advance; they only need to express their operation intention (such as "remove this screw"), and detailed step guidance will be automatically generated. For non-professional users, this enables them to complete complex operations that originally required professional knowledge and experience.

[0115] Operation instructions refer to user-issued commands that instruct the system to perform specific operations on the operated parts. Operation instructions clarify the user's intent, thereby determining the specific operational steps. Operation instructions can be obtained through voice input, interface selection, gesture recognition, automatic inference, and preset instructions. For example, automatic inference can infer the user's intent based on contextual information; for instance, when a user aligns a tool with a component and applies rotational force, it can be automatically inferred that the user is performing a disassembly operation. Preset instructions, on the other hand, can be pre-defined in specific application scenarios; for example, in a repair process, the next operation instruction can be automatically determined based on the current repair step.

[0116] A target operation can refer to the specific type of operation that a user intends to perform on the part being operated on. For example, target operations can include disassembly operations, installation operations, testing operations, adjustment operations, maintenance operations, etc.

[0117] A model can refer to an intelligent model used to transform operational intentions and component information into a sequence of specific operational steps. For example, a model can be a rule-based model, a machine learning-based model, a knowledge graph-based model, or a physical simulation-based model, etc.

[0118] In some embodiments, operating instructions and identification information of the operated part can be used as input to the model. Identification information may include part name, specifications, location information, and connection relationships with other parts. The model infers based on the input information to generate a sequence of operating steps. For large language models, carefully designed prompts can guide the model to output structured operating steps. Then, the generated step sequence is validated to ensure the completeness, logical order, and feasibility of the steps. Finally, based on parameters in the part identification information, such as screw specifications and torque requirements, the specific operating parameters in the steps are filled in.

[0119] In some embodiments, the operation guidance data can be information content generated based on a sequence of operation steps to guide the user in completing the target operation. Examples include pre-operation preparation guidance such as tool selection and component positioning; real-time guidance during operation such as direction indicators, force prompts, and progress feedback; and post-operation verification guidance such as checking and confirming, and testing and verifying.

[0120] In some embodiments, the operation guidance data can be determined by directly converting each step in the operation sequence into corresponding guidance content, forming sequential guidance data. Alternatively, a preset guidance template can be used, with parameters from the step sequence filled into the template to generate uniformly formatted guidance data. Or, the most suitable guidance format can be dynamically generated based on the characteristics of the step sequence and the current operation stage. For example, visual instructions can be generated for positioning steps, and numerical displays can be generated for parameter-related steps.

[0121] According to embodiments of this application, physical interaction state information generated by the operating tool when performing a target operation can be obtained; the physical interaction state information is compared with the expected state of the current step in the operation step sequence to obtain a comparison result; and the operation guidance data is updated based on the comparison result. Users no longer need to discover errors only after completing the operation, but can obtain instant feedback during the operation.

[0122] Physical interaction state information refers to the dynamic parameter data collected in real time by various sensors integrated on the tool during the physical interaction between the tool and the operated part. For example, physical interaction state information can include mechanical parameters, kinematic parameters, time parameters, positional parameters, and state parameters. The tool generates this information when performing the target operation, meaning that information collection and the operation process are synchronized. For instance, when a user uses a smart screwdriver to tighten a screw, a torque sensor continuously collects torque values ​​at millisecond frequencies, and an inertial measurement unit collects rotation angles and angular velocities. This data can constitute a continuous state information stream reflecting the operation process.

[0123] In some embodiments, the expected state can refer to the target state or condition that should be achieved to complete the current step. For example, the expected state may include target values, threshold ranges, state conditions, timing requirements, etc. The expected state can be predefined in the sequence of operation steps and associated with each step. For example, for the "tightening screws" step, the expected state can be defined as "torque reaches 1.5 N·m" or "rotation angle reaches 90° and torque is stable".

[0124] In some embodiments, comparison can refer to comparing the real-time collected physical interaction state information with the expected state of the current step to determine whether the operation meets expectations. For example, comparison can include numerical comparison, state matching, trend analysis, pattern recognition, etc.

[0125] For example, numerical comparison can be to compare the real-time torque value with the target torque value, calculate the difference, or determine whether a threshold has been reached. For instance, if the current torque value is 1.2 N·m and the target torque value is 1.5 N·m, the comparison result would be "the target has not been reached, and further action is required."

[0126] For example, state matching can be used to match the real-time state with the expected state conditions and determine whether they are met. For instance, to determine whether "the screw has been fully screwed in", the torque change curve can be analyzed, and when the torque changes from rising to stabilizing, it can be determined that the screwing is complete.

[0127] For example, trend analysis can be used to analyze the changing trends of status information and determine whether the operation is progressing in the expected direction. For instance, analyzing the rate of torque increase can help determine whether the tightening process is normal.

[0128] For example, pattern recognition can use machine learning models to identify sequences of state information and determine whether the current operation conforms to the expected pattern.

[0129] In some embodiments, updating operation guidance data may include progress feedback, operation confirmation, error correction prompts, early warning prompts, error alarms, parameter adjustments, and step jump waiting. For example, parameter adjustment refers to dynamically adjusting guidance parameters according to actual conditions, such as estimating the required rotation angle based on the current torque increase rate and dynamically updating the target indication. For example, step jump may be skipping or reverting certain steps under specific conditions, such as automatically skipping the "initial loosening" step if a screw is detected to be loose.

[0130] Based on the above data processing method, embodiments of this application also provide an intelligent operating tool and a head-mounted display device. The following, in conjunction with... Figures 6-7 The intelligent operating tool and head-mounted display device are illustrated.

[0131] Figure 6 A schematic diagram of the structure of the intelligent operating tool according to an embodiment of this application is shown.

[0132] like Figure 6 As shown, the tool 600 includes a tool body 610, a sensing module 620 integrated into the tool body 610, and a processing and communication module 630 coupled to the sensing module. The sensing module 620 acquires local perspective information of the object being manipulated when the tool body 610 points at it. This local perspective information focuses on the microscopic details of the interaction between the tool and the object. After acquiring this local perspective information, the processing and communication module 630 processes it using a model to identify and determine the identifier information of the manipulated part within the object. Subsequently, the processing and communication module 630 sends this identifier information to a terminal device, which then completes the subsequent determination of operation guidance data and generation of display information.

[0133] The intelligent operation tool provided in this application's embodiments achieves edge intelligence by embedding component recognition functionality at the tool's end. This means the tool itself possesses recognition capabilities, outputting only semantically defined identifier information rather than raw image data. This design reduces reliance on network bandwidth, as the identifier data volume is significantly smaller than the original video stream, making it suitable for use in bandwidth-constrained environments. Furthermore, it protects visual privacy at the operation site, as the original image remains local to the tool. Finally, it establishes a clear functional division between the tool and terminal devices—the tool handles perception and recognition, while the terminal handles presentation and interaction, creating an efficient edge-cloud collaborative architecture. This tool can be used as an independent product with various terminal devices, providing a standardized intelligent perception entry point for scenarios such as product design review, equipment maintenance, and user self-service operations.

[0134] According to an embodiment of this application, the processing and communication module 630 is further configured to acquire interaction information when the operating tool physically interacts with the operated part, determine the current stage of the physical interaction based on the interaction information, determine corresponding stage guidance information from the operation guidance data based on the current stage, and send the stage guidance information and the identification information to the terminal device so that the terminal device updates the display information according to the stage guidance information. Compared with the scheme of delegating the original interaction information or stage judgment task to the terminal device, this embodiment completes the stage judgment and local extraction of guidance information at the tool end, and only sends lightweight stage guidance information to the terminal device. This can reduce the amount of data transmission between the tool and the terminal device, and can be applied to application scenarios where wireless communication bandwidth is limited or low-power operation is required.

[0135] Figure 7 A schematic diagram of a head-mounted display device according to an embodiment of this application is shown.

[0136] like Figure 7As shown, the head-mounted display device 700 includes a communication module 710, a processing module 720, and a display module 730.

[0137] The communication module 710 is configured to receive target identification information from the intelligent operating tool, the target identification information being determined by the intelligent operating tool using a model based on the collected local perspective information of the operating object.

[0138] In some embodiments, the communication module may use Bluetooth, Wi-Fi, UWB or a dedicated short-range communication protocol to establish a point-to-point connection with the smart operating tool.

[0139] In some embodiments, the received identification information may be a semantic result that has already been processed by the tool, rather than the original image data. This design allows the head-mounted display device to function without image recognition capabilities, only needing to process lightweight identification data, thus reducing the device's hardware complexity and power consumption.

[0140] In some embodiments, the communication between the communication module and the tool can be one-way reception or a two-way confirmation mechanism can be established to ensure the reliability of the identification information transmission.

[0141] The processing module 720 is coupled to the communication module to acquire operation guidance data associated with the target identification information; and to generate display information, which is used to overlay the operation guidance data on the spatial position corresponding to the operation part indicated by the target identification information in the user's first view.

[0142] In some embodiments, obtaining operation guidance data associated with target identification information can be achieved in the following ways: querying the operation guidance data corresponding to the identification information from the device's local cache database; initiating a request to the cloud server via the communication module to obtain real-time updated operation guidance content; or directly parsing the operation guidance data from the extended data packet received by the smart operation tool. The source of the operation guidance data can be a pre-built product maintenance manual library, a cloud knowledge base, or auxiliary information uploaded by the tool.

[0143] The display module 730 is used to output the display information.

[0144] In some embodiments, the specific implementation of the display module can be an optical perspective display, which projects virtual content onto a semi-transparent mirror using a micro-projector, and the content enters the human eye after being superimposed with real ambient light; or it can be a video perspective display, which renders a virtual-real fusion image on a micro-display screen, and the user observes it indirectly through an environmental image captured by a camera.

[0145] In some embodiments, the display module may further include a binocular display unit to present stereoscopic virtual content with depth information, further enhancing the accuracy of the user's perception of the spatial position of the operating part.

[0146] According to embodiments of this application, any plurality of modules among the communication module 710, processing module 720, and display module 730 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the acquisition module, generation module, and determination module can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the acquisition module, generation module, and determination module can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0147] Figure 8 A schematic block diagram of an example electronic device that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (including augmented reality glasses, mixed reality headsets, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0148] like Figure 8As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the device 800, including but not limited to operation guidance data, component identification information mapping tables, identification model parameters, augmented reality rendering data, etc. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0149] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, gesture recognition sensor, voice input device, etc.; output unit 807, such as various types of monitors, speakers, augmented reality display devices, etc.; storage unit 808, such as disk, optical disk, solid-state drive, etc.; and communication unit 809, such as network card, modem, wireless communication transceiver (including Bluetooth, Wi-Fi, 5G, UWB, etc.). Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, such as receiving first-view information from head-mounted sensing devices, receiving partial-view information and interactive information from smart operating tools, and sending composite images to display terminals, etc.

[0150] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), a neural network processor (NPU), a field-programmable gate array (FPGA), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data information processing methods. In some embodiments, the data information processing method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data information processing method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a data information processing method by any other suitable means (e.g., by means of firmware).

[0151] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code used to implement the methods of this application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Python, Java, C++) and traditional procedural programming languages ​​(such as C). This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, augmented reality display device, head-mounted display, etc.); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. For augmented reality and wearable device scenarios, interaction methods such as gesture recognition, voice input, and eye tracking can also be used. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, tactile input, and gesture input).

[0155] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers, head-mounted displays, smart operating tools, etc., with a graphical user interface or web browser through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, mobile communication networks, satellite communication networks, etc.

[0156] A computer system may include clients and servers. Clients and servers are generally located far apart and typically interact via a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. In the embodiments of this application, head-mounted sensing devices, smart operating tools, processing devices, remote terminals, etc., can act as either clients or servers, flexibly configured according to specific application scenarios. The server can be a cloud server, a distributed system server, or a server incorporating blockchain technology.

[0157] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0158] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data information processing method, comprising: Acquire a first signal, the first signal being from a tool-end sensing device integrated into the operating tool, the first signal including at least local view information collected by the sensing device and pointing to the operating object; Based on the local perspective information, the identification information of the operated part in the operation object is determined using the model; Determine the operational guidance data associated with the identification information; Generate display information, which is used to overlay the operation guidance data onto a spatial position corresponding to the operated part through a display interface.

2. The method according to claim 1, wherein, The first signal also includes interaction information characterizing the physical interaction between the operating tool and the operated part; The method further includes: Based on the interaction information, determine the current stage of the physical interaction; Based on the current stage, determine the corresponding stage guidance information from the operation guidance data; Update the stage guidance information to the display information.

3. The method according to claim 1, wherein, The method further includes: Acquire a second signal, the second signal being from a head-mounted sensing device, the second signal including at least a first perspective information characterizing the user's view of the operating environment containing the operating object; The generated display information includes: Based on the first perspective information, the display information is generated so that the operation guidance data is embedded in the first perspective information at the position corresponding to the operated part.

4. The method according to claim 3, wherein, The method further includes: Based on the displayed information, the first perspective information is processed to generate a first image; The first image is sent to a display terminal so that the display terminal displays the first image.

5. The method according to claim 4, wherein, The method further includes: A second image is generated based on the local perspective information; A multi-view interface is generated, which is configured to simultaneously display the first screen and the second screen; In response to a perspective switching command triggered on the multi-view interface, the main display content of the multi-view interface is switched between the first screen and the second screen.

6. The method according to claim 1, wherein, The operational guidance data associated with the identification information includes: Receive an operation instruction to perform a target operation on the operated part; Based on the operation instructions and the identification information, the sequence of operation steps is determined using a model; Based on the sequence of operation steps, the operation guidance data is determined, and the operation guidance data is used at least to guide the user to complete the target operation.

7. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs; The one or more processors execute the one or more computer programs to perform the following operations: Acquire a first signal, the first signal being from a tool-end sensing device integrated into the operating tool, the first signal including at least local view information collected by the sensing device and pointing to the operating object; Based on the local perspective information, the identification information of the operated part in the operation object is determined using the model; Determine the operational guidance data associated with the identification information; Generate display information, which is used to overlay the operation guidance data onto a spatial position corresponding to the operated part through a display interface.

8. An intelligent operation tool, comprising: Tool body; The sensing module, integrated into the tool body, is used to collect local view information of the operation object when the tool body is pointing at at least one operation object; The processing and communication module, coupled to the sensing module, is configured as follows: Obtain the local viewpoint information; Based on the local perspective information, the identification information of the operated part in the operation object is determined using the model; The identification information is sent to the terminal device so that the terminal device can determine the operation guidance data associated with the identification information and generate display information. The display information is used to overlay the operation guidance data in the spatial position corresponding to the operated part in the user's first perspective.

9. The intelligent operating tool according to claim 8, wherein, The processing and communication module is further configured to: Acquire interaction information when the operating tool physically interacts with the operated part; Based on the interaction information, determine the current stage of the physical interaction; Based on the current stage, determine the corresponding stage guidance information from the operation guidance data; The stage guidance information and the identification information are sent to the terminal device so that the terminal device updates the display information according to the stage guidance information.

10. A head-mounted display device, comprising: The communication module is configured to receive target identification information from the intelligent operation tool, wherein the target identification information is determined by the intelligent operation tool using a model based on the collected local view information of the operation object; The processing module, coupled to the communication module, acquires operation guidance data associated with the target identification information; Generate display information, which is used to overlay the operation guidance data onto the spatial position in the user's first-person view corresponding to the operation part indicated by the target identification information; The display module is used to output the display information.

Images