Non-contact processing equipment interaction method and interaction system, storage medium and head-mounted intelligent device

Abstract

The invention provides a non-contact processing equipment interaction method, which is applied to a head-mounted intelligent device, and comprises the following steps: acquiring an equipment scene image obtained based on a scene of actual processing equipment in front of eyes of a user, the equipment scene image comprising a processing equipment image and a selection gesture image of the user formed in the processing equipment image; acquiring eye state information of a user; identifying a processing equipment image selected by the user according to the eye state information and the selection gesture image; based on the processing equipment image selected by the user, displaying a selection success identifier, and determining the processing equipment image selected by the user as actual processing equipment; and obtaining operation parameters of the selected actual processing equipment, and displaying an operation parameter interface of the selected actual processing equipment. According to the interaction method provided by the invention, a user can quickly acquire the operation parameters of the processing equipment. The invention further provides a storage medium, a head-mounted intelligent device and a non-contact processing equipment interaction system.

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology, and particularly to a non-contact processing equipment interaction method, a storage medium storing the method, a head-mounted intelligent device applying the method, and a non-contact processing equipment interaction system. Background Art

[0002] Currently, intelligent wearable electronic devices display pre-shot panoramic pictures or videos to users through relevant technologies, such as panoramic picture or video image pages without blind spots in all directions, such as workshops, indoors, outdoors, and their equipment, products, personnel, service operations, etc. The panoramic image page can only display pre-shot and pre-produced image materials and cannot perform real-time operations on the equipment running in the workshop. Summary of the Invention

[0003] In view of the above, it is necessary to provide a non-contact processing equipment interaction method to facilitate quick identification of a selected processing equipment and obtain the operating parameters of the selected processing equipment.

[0004] On the one hand, the present application provides a non-contact processing equipment interaction method applied to a head-mounted intelligent device, which includes: Obtaining an equipment scene image based on the scene of the actual processing equipment in front of the user, the equipment scene image including a processing equipment image and a selection gesture image of the user formed in the processing equipment image; Obtaining the eye state information of the user; Identifying the processing equipment image selected by the user according to the eye state information and the selection gesture image; Based on the processing equipment image selected by the user, displaying a selection success mark and determining the processing equipment image selected by the user as the selected actual processing equipment; Obtaining the operating parameters of the selected actual processing equipment and displaying an operating parameter interface of the selected actual processing equipment.

[0005] The non-contact processing equipment interaction method provided by the present application can obtain the scene image of the processing equipment by obtaining the scene of the actual processing equipment in front of the user. By obtaining the eye state information of the user and the selection gesture image of the user, and combining with the scene image, the processing equipment selected by the user can be identified, and the operating parameter interface of the processing equipment selected by the user can be displayed to the user, so as to facilitate the user to obtain the operating parameter information of the selected processing equipment, realize the interaction with the processing equipment in a non-contact state, and is beneficial to quickly and conveniently determining the processing equipment selected by the user when facing multiple processing equipment, thereby improving the experience of non-contact interaction.

[0006] In one embodiment, the selected gesture image has a gesture circle. The steps of identifying the processing device image selected by the user according to the eye state information and the selected gesture image include: displaying a reference circle in the spatial interface; respectively and one-to-one determining the pupil center, the reference center, and the gesture center based on the eye state information, the reference circle, and the gesture circle; judging whether the pupil center, the reference center, and the gesture center are on a preset condition straight line; if the pupil center, the reference center, and the gesture center are on the preset condition straight line, determining the device scene image within the gesture circle as the gesture circle device image; judging whether there is a preset object in the gesture circle device image; and determining the preset object within the gesture circle device image as the processing device image selected by the user.

[0007] In one embodiment, the step of displaying a selection success mark includes: in the spatial interface, switching the color of the reference circle to form the selection success mark.

[0008] In one embodiment, the non-contact processing device interaction method further includes: acquiring a parameter retrieval gesture image of the user; judging whether the parameter retrieval gesture image has the gesture circle; if the parameter retrieval gesture image has the gesture circle, judging whether there is a finger gesture extended on the gesture circle in the parameter retrieval gesture image; if there is a finger gesture extended on the gesture circle in the parameter retrieval gesture image, judging whether the finger gesture in the parameter retrieval gesture image matches a preset gesture; and if the finger gesture in the parameter retrieval gesture image matches the preset gesture, executing the step of "acquiring the operating parameters of the actually selected processing device".

[0009] In one embodiment, the non-contact processing device interaction method further includes: judging whether the number of the processing device images selected by the user meets a preset number; if the number of the processing device images selected by the user meets the preset number, executing the step of "displaying a selection success mark based on the processing device image selected by the user, and determining one of the processing device images selected by the user as the actually selected processing device".

[0010] In one embodiment, the non-contact processing device interaction method further includes: acquiring the voice signal of the user; identifying a confirmation parameter retrieval voice in the voice signal; and based on the confirmation parameter retrieval voice, executing the step of "acquiring the operating parameters of the actually selected processing device".

[0011] In one embodiment, the non-contact processing equipment interaction method further includes: if multiple processing devices are selected by the user, switching the color of the reference reference circle on the spatial digital interface to form a prompt identifier.

[0012] In one embodiment, the non-contact processing equipment interaction method further includes: acquiring an operation action image of the user based on the operation parameter interface; correspondingly controlling the selected actual processing equipment according to the operation action image.

[0013] In one embodiment, the non-contact processing equipment interaction method further includes: scanning the scene of the actual processing equipment to form the equipment scene image, wherein the actual processing equipment has an identification diagram; constructing a virtual reality scene with virtual processing equipment based on the equipment scene image; identifying the identification code of each actual processing equipment based on the identification diagram of each actual processing equipment in the equipment scene image; in the virtual reality scene, calibrating the identification code of each actual processing equipment to each virtual processing equipment one by one, so as to associate the operation parameters of the actual processing equipment with the virtual processing equipment based on the identification code.

[0014] In one embodiment, the non-contact processing equipment interaction method further includes: scanning the scene of the actual processing equipment to form the equipment scene image; constructing a virtual reality scene with virtual processing equipment based on the equipment scene image; sending the virtual reality scene to the server; receiving the calibrated virtual reality scene feedback by the server, wherein each virtual processing equipment in the calibrated virtual reality scene is calibrated with an identification code, and the identification code calibrated for each virtual processing equipment corresponds to the identification code of each actual processing equipment one by one, so as to associate the operation parameters of the actual processing equipment with the virtual processing equipment based on the identification code.

[0015] In one embodiment, the step of identifying the processing equipment image selected by the user according to the eye state information and the selection gesture image includes: determining the spatial position of the user's eyes based on the eye state information and the virtual reality scene; determining the spatial position of the equipment selection gesture based on the selection gesture image and the virtual reality scene; connecting the spatial position of the user's eyes and the spatial position of the equipment selection gesture in series to form a spatial straight line based on the virtual reality scene; identifying whether there is a virtual processing equipment on the spatial straight line based on the virtual reality scene; determining the virtual processing equipment on the spatial straight line as the processing equipment image selected by the user.

[0016] In one embodiment, the non-contact processing equipment interaction method further includes: establishing a communication connection with the actual processing equipment based on the selected actual processing equipment.

[0017] In one embodiment, the non-contact processing equipment interaction method further includes: based on the virtual reality scene, displaying an operation parameter interface of the selected actual processing equipment on one side of the virtual processing equipment; or, based on the virtual reality scene, hiding the virtual processing equipment and displaying the operation parameter interface of the actual processing equipment on one side of the actual processing equipment.

[0018] The present application also provides a storage medium for storing a computer program or code, which, when executed by a processor, implements the above non-contact processing equipment interaction method.

[0019] The present application also provides a head-mounted intelligent device, including: A camera module for obtaining an equipment scene image based on the scene of the actual processing equipment in front of the user's eyes, the equipment scene image including a processing equipment image and a selection gesture image of the user formed in the processing equipment image; An eye movement tracking module for obtaining the eye state information of the user; A processor for identifying the processing equipment image selected by the user according to the eye state information and the selection gesture image; A display module for displaying a selection success mark based on the processing equipment image selected by the user; The processor is further configured to determine one of the processing equipment images selected by the user as the selected actual processing equipment; and to obtain the operation parameters of the selected actual processing equipment; The display module is further configured to display an operation parameter interface of the selected actual processing equipment.

[0020] With the head-mounted intelligent device provided by the present application, by setting a camera module, the scene of the actual processing equipment in front of the user can be obtained to obtain a scene image of the processing equipment, and the selection gesture image of the user can also be obtained. By setting an eye movement tracking module, the eye state information of the user can be obtained. By setting a processor, the processing equipment selected by the user can be identified by combining the scene image, the selection gesture image and the eye state information. By setting a display module, the operation parameter interface of the processing equipment selected by the user can be displayed to the user, so as to facilitate the user to obtain the operation parameter information of the selected processing equipment, which is beneficial to quickly and conveniently determine the processing equipment selected by the user when facing multiple processing equipment, thereby improving the non-contact interaction experience.

[0021] The present application also provides a non-contact processing equipment interaction system, which includes: An actual processing equipment; A head-mounted intelligent device, configured to: Obtain an equipment scene image formed based on the scene of the actual processing equipment in front of the user, where the equipment scene image includes a processing equipment image and a selection gesture image of the user formed in the processing equipment image; Obtain the eye state information of the user; Identify the processing equipment image selected by the user according to the eye state information and the selection gesture image; Based on the processing equipment image selected by the user, display a selection success mark, and determine one of the processing equipment images selected by the user as the selected actual processing equipment; Obtain the operation parameters of the selected actual processing equipment, and display an operation parameter interface of the selected actual processing equipment; A server, configured to construct a virtual reality scene based on the scene of the actual processing equipment, where each virtual processing equipment in the virtual reality scene corresponds to each actual processing equipment one by one, and the server is configured to associate the operation parameters of the selected actual processing equipment with the corresponding virtual processing equipment, so that the corresponding virtual processing equipment is associated with the operation parameters; A remote terminal, configured to obtain the virtual reality scene, and display the corresponding virtual processing equipment and the associated operation parameters.

[0022] In the non-contact processing equipment interaction system provided by the present application, by setting a head-mounted intelligent device, the scene of the actual processing equipment in front of the user can be obtained to obtain a scene image of the processing equipment, and the eye state information of the user and the selection gesture image of the user can be obtained. Combining with the scene image, the processing equipment selected by the user can be identified, and the operation parameter interface of the processing equipment selected by the user can be displayed to the user, so as to facilitate the user to obtain the operation parameter information of the selected processing equipment, realize the interaction with the processing equipment in a non-contact state, and is beneficial to quickly and conveniently determine the processing equipment selected by the user when facing multiple processing equipment, thereby improving the experience of non-contact interaction. By setting a server and a remote terminal, a virtual reality scene can be constructed, which is thus associated with the parameters of the actual processing equipment, facilitating the remote acquisition of the operation parameters of the processing equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic structural diagram of the non-contact processing equipment interaction system provided by an embodiment of the present application.

[0024] Figure 2Schematic diagram of the head-mounted intelligent device provided by the embodiment of the present application.

[0025] Figure 3 Schematic diagram of the interface for displaying the operating parameters of the head-mounted intelligent device in the embodiment of the present application.

[0026] Figure 4 Schematic diagram of the modules of the head-mounted intelligent device in the embodiment of the present application.

[0027] Figure 5 Flowchart of the non-contact processing equipment interaction method provided by the embodiment of the present application.

[0028] Figure 6 Schematic diagram of the selection state of the non-contact processing equipment interaction system provided by the embodiment of the present application.

[0029] Figure 7 Schematic diagram of the prompt state of the non-contact processing equipment interaction system provided by the embodiment of the present application.

[0030] Figure 8 Schematic diagram of the confirmation state of the non-contact processing equipment interaction system provided by the embodiment of the present application.

[0031] Description of the main component symbols Non-contact processing equipment interaction system: 100 Head-mounted intelligent device: 10 Housing: 11 Camera module: 12 Eye movement tracking module: 13 Display module: 14 Operating parameter interface: 141 Reference benchmark circle: 142 Selection success mark: 143 Prompt mark: 144 Processor: 15 Sound sensor: 16 Processing equipment: 20a, 20b, 20c Server: 30 Eye: E Pupil center: E1 Gesture circle: H Gesture center: H1 Finger gesture: H2 Preset condition line: L Steps: S1, S2, S3, S4, S5.

[0032] The following specific embodiments will further illustrate the present application in conjunction with the above drawings. Specific embodiments

[0033] Next, in combination with the accompanying drawings in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments.

[0034] Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application.

[0035] In order to further elaborate on the technical means and effects adopted by the present application to achieve the predetermined purpose, the following detailed description of the present application is made in combination with the accompanying drawings and preferred embodiments.

[0036] Please refer to Figure 1 , the non-contact processing equipment interaction system 100 provided by the embodiments of the present application includes a head-mounted intelligent device 10 and a plurality of actual processing devices, such as Figure 1 the shown processing device 20a, processing device 20b, and processing device 20c. Among them, the head-mounted intelligent device 10 is used for the user to wear and display the operating parameters of the processing device selected by the user to the user. The head-mounted intelligent device 10 can be a smart glasses, a head-mounted display, a smart helmet, etc. Specifically, the head-mounted intelligent device 10 is used to obtain the image of the direction the user is facing, so as to obtain the gesture image of the user and the image of the processing device. The head-mounted intelligent device 10 is also used to obtain the eye state information of the user, so as to identify the processing device selected by the user according to the gesture image, the processing device image, and the eye state information, and obtain the operating parameters of the processing device selected by the user, so as to display the operating parameter interface of the processing device selected by the user.

[0037] The non-contact processing equipment interaction system 100 further includes a server 30, and the server 30 is communicatively connected to a plurality of processing devices and the head-mounted intelligent device 10. The server 30 is used to construct a virtual reality scene based on the actual processing equipment scene. The virtual reality scene includes a plurality of virtual processing devices, and each virtual processing device corresponds to each actual processing device one by one. The server 30 is also used to associate the operating parameters of the selected processing device with the corresponding virtual processing device, so that the virtual processing devices in the virtual reality scene can be associated with the operating parameters. Thus, when a remote terminal accesses the server 30, the virtual reality scene can be obtained, and the corresponding virtual processing device and its associated operating parameters can be displayed.

[0038] The remote terminal can be a mobile phone, a computer, a tablet, a mobile phone, a central control platform, etc.

[0039] The non-contact processing equipment interaction system 100 provided by the embodiments of the present application can be connected to the processing equipment without contacting the processing equipment by setting the head-mounted intelligent device 10, and display the operating parameters of the processing equipment to the user, which is beneficial for the user to conveniently and intuitively view the operating parameters of the processing equipment, thereby improving the efficiency and convenience of viewing the equipment. By setting the head-mounted intelligent device 10 to identify the processing equipment selected by the user, the operating parameters of the selected processing equipment can be viewed specifically, so that the picture in the user's field of view is simple and the accuracy of observation is improved. By setting the server 30, the actual processing equipment in the actual scene can be synchronously associated with the virtual processing equipment in the virtual reality scene, so that the virtual processing equipment can be associated with the operating parameters of the actual processing equipment, which is beneficial for obtaining the operating parameters of the selected processing equipment when accessing the server 30 through a remote terminal.

[0040] Please refer to Figure 2 , Figure 2 As an example of the head-mounted intelligent device 10, it includes a housing 11, a camera module 12, an eye movement tracking module 13, and a display module 14. The camera module 12, the eye movement tracking module 13, and the display module 14 are all arranged on the housing 11. The housing 11 is used to cover and fix on the user's head, so that the user's eyes correspond to the eye movement tracking module 13 and the display module 14, and the camera module 12 faces the direction where the user's eyes are looking.

[0041] The camera module 12 is used to obtain images of the direction the user is facing, so as to obtain images of the processing equipment when the user looks at the processing equipment, and obtain images of gesture actions when the user makes gesture actions. The camera module 12 includes two cameras arranged at intervals, and each camera respectively obtains corresponding images, so that the camera module 12 can realize binocular vision function, and thus the depth information of the scene towards which the camera module 12 is facing can be calculated through the images respectively obtained by the two cameras, and then the three-dimensional space information of the scene towards which the camera module 12 is facing can be determined. Of course, in other embodiments, the number of cameras can also be one or more than two, which can be set according to actual needs.

[0042] The eye movement tracking module 13 is used to obtain the eye state information of the user. Specifically, the eye movement tracking module 13 is used to obtain the optical information of the eyes, so as to determine the pupil center of the eyes, and then confirm the gaze direction of the eyes.

[0043] The display module 14 makes the space interface in front of the user's eyes display information. Specifically, the display module 14 is an augmented reality (AR) display module, that is, the display module 14 superimposes and displays on the basis of the user seeing the real-world scene.

[0044] Please refer to Figure 3 , for example, the display module 14 can display the operation parameter interface 141 of the processing device 20b on the spatial interface, so as to facilitate the user to intuitively view the operation parameters of the processing device 20b. The operation parameter interface 141 can cover the display of the processing device 20b, or can be displayed above or on the side of the processing device 20b to avoid blocking the processing device 20b, and this application does not limit this.

[0045] Please refer to Figure 4 , the head-mounted intelligent device 10 further includes a processor 15 and a sound sensor 16. The processor 15 is electrically connected to the camera module 12, the eye movement tracking module 13, the display module 14, and the sound sensor 16. The processor 15 is used to receive the image information captured by the camera module 12 and the eye state information of the user obtained by the eye movement tracking module 13, so as to identify the processing device selected by the user. The processor 15 is also used to obtain the operation parameters of the selected processing device, thereby generating an image of the operation parameter interface and transmitting the image of the operation parameter interface to the display module 14.

[0046] The sound sensor 16 is used to obtain the sound signal of the user and transmit the sound signal to the processor 15. The processor 15 is also used to identify the voice information in the sound signal, so as to perform corresponding operations. For example, the voice signal may include a voice for confirming parameter retrieval, which is used to confirm the selected processing device by voice during the process of selecting the processing device. The voice signal may also include a control voice, which is used to remotely control the selected processing device. The setting of the voice can be set according to actual needs, and this application does not limit this.

[0047] Please refer to Figure 5 , the embodiment of this application also provides a non-contact processing device interaction method applied to the above-mentioned head-mounted intelligent device 10, which includes: Step S1: Obtain a device scene image based on the scene of the actual processing device in front of the user's eyes, and the device scene image includes a processing device image and a selection gesture image of the user formed in the processing device image.

[0048] Specifically, step S1 is performed by the camera module 12 of the head-mounted intelligent device 10. The two cameras of the camera module 12 simultaneously collect the scene images of the actual processing device and transmit them to the processor 15. The processor 15 constructs a device scene image with three-dimensional information according to the binocular vision principle and the scene images collected by the two cameras. The device scene image includes the three-dimensional information of the actual processing device and the three-dimensional information of the user's selection gesture.

[0049] In an embodiment, before step S1, the non-contact processing device interaction method further includes: Step S01: Scan the scene of the actual processing equipment to form an equipment scene image, where the actual processing equipment has an identification diagram; Specifically, step S01 can be scanned by the camera module 12 of the head-mounted intelligent device 10. The identification diagram can be a pattern set on the processing equipment, such as a QR code, a barcode, etc., and the present application does not limit this.

[0050] Step S02: Based on the equipment scene image, construct a virtual reality scene with virtual processing equipment; Specifically, the processor 15 of the head-mounted intelligent device 10 can obtain the three-dimensional space information of the scene where the processing equipment is located according to the equipment scene image obtained by the camera module 12, so as to construct a virtual reality scene according to the three-dimensional space information and construct a virtual processing equipment corresponding to the actual processing equipment.

[0051] Step S03: Based on the identification diagram of each actual processing equipment in the equipment scene image, identify the identification code of each actual processing equipment; Step S04: In the virtual reality scene, calibrate the identification code of each actual processing equipment to each virtual processing equipment one by one, so as to associate the operation parameters of the actual processing equipment with the virtual processing equipment based on the identification code.

[0052] Specifically, by confirming the identification diagram on the actual processing equipment, each processing equipment can be identified. Since the virtual processing equipment in the virtual reality scene corresponds to the actual processing equipment in the actual scene one by one, the identification code of the actual processing equipment can be calibrated to the virtual processing equipment one by one, so that the virtual processing equipment is associated with the actual processing equipment. Step S04 can be directly executed by the head-mounted intelligent device 10, or can be executed by the server 30 and then the head-mounted intelligent device 10 downloads the calibrated virtual reality scene from the server 30 for identifying the processing equipment selected by the user.

[0053] In another embodiment, before step S1, the non-contact processing equipment interaction method further includes: Step S11: Scan the scene of the actual processing equipment to form the equipment scene image; Step S12: Based on the equipment scene image, construct a virtual reality scene with virtual processing equipment; Step S13: Send the virtual reality scene to the server; Step S14: Receive the calibrated virtual reality scene fed back by the server.

[0054] Among them, each of the virtual processing devices in the calibrated virtual reality scene is calibrated with an identification code, and the identification code calibrated for each virtual processing device corresponds one-to-one with the identification code of each actual processing device, so as to associate the operating parameters of the actual processing device with the virtual processing device based on the identification code.

[0055] Specifically, the calibration of the virtual processing device can be performed in the server 30. The virtual processing device in the virtual reality scene can be calibrated manually by the operator, or the server 30 can match the virtual processing device in the virtual reality scene according to the pre-recorded information such as the position and appearance of the processing device, so as to calibrate the virtual processing device. The head-mounted intelligent device 10 can download the calibrated virtual reality scene from the server 30, so as to determine the processing device selected by the user.

[0056] Step S2: Obtain the eye state information of the user; Specifically, step S2 is executed by the eye movement tracking module 13 of the head-mounted intelligent device 10, which is used to capture the image of the eye and its reflection, analyze the three-dimensional model of the eye and the position of the eyeball in space through an algorithm to determine the pupil center and corneal reflection, and further calculate the position of the fixation point. The eye state information may include the image of the eye and its reflection.

[0057] Step S3: Identify the image of the processing device selected by the user according to the eye state information and the selection gesture image; Please refer to Figure 6 , in this embodiment, the selection gesture of the user is specifically to make a virtual circle by holding the thumb and the other four fingers together, so as to form a gesture circle H through which the user's line of sight can pass.

[0058] Step S3 specifically includes: Step S31: Display a reference reference circle on the space interface; Specifically, in this embodiment, the reference reference circle 142 is a white circle projected by the display module 14 in front of the user, and the reference reference circle 142 is used to assist the user in determining the direction of fixation and the placement position of the gesture.

[0059] Step S32: Based on the eye state information, the reference reference circle, and the gesture circle, respectively determine the pupil center, the reference reference center, and the gesture center in one-to-one correspondence; Specifically, the pupil center E1 is the spatial position where the pupil of the user's eye E is located obtained by the eye movement tracking module 13 according to the eye state information of the user. The reference reference center 1421 is the geometric center of the reference reference circle 142. The gesture center H1 is the geometric center of the gesture circle H.

[0060] Before performing step S32, it further includes: Step S321: Obtain the gesture image of the user; Step S322: Determine whether the gesture image has the gesture circle. If it has the gesture circle, perform step S32; Specifically, multiple gesture images with a gesture circle H are pre-stored in the processor 15 of the head-mounted intelligent device 10. After the camera module 12 obtains the gesture image of the user, by comparing it with the gesture images stored in the processor 15 and using a deep learning algorithm, it is determined whether the gesture image of the user has the gesture circle H.

[0061] Step S33: Determine whether the pupil center, the reference benchmark center, and the gesture center are on a preset condition straight line; Specifically, the preset condition straight line L is the straight line defined by connecting any two of the pupil center E1, the reference benchmark center 1421, and the gesture center H1. Step S33 specifically first selects any two points from the pupil center E1, the reference benchmark center 1421, and the gesture center H1 to construct the preset condition straight line L, and then determines whether the other point is located on the preset condition straight line L or within a certain distance from the preset condition straight line L.

[0062] Step S34: If the pupil center, the reference benchmark center, and the gesture center are on the preset condition straight line, determine the device scene image within the gesture circle as the gesture circle device image; Specifically, in step S34, the processor 15 first determines the spatial position of the gesture circle H according to the device scene image with three-dimensional information collected by the camera module 12, and then determines the spatial position of the user's eyes E according to the eye movement tracking module 13. After determining the positions of the gesture circle H and the eyes E, the processor 15 can determine the range of the scene seen by the user's eyes E through the gesture circle H along the edge of the eyes E and the gesture circle H, and combine it with the device scene image to determine the gesture circle device image.

[0063] Step S35: Determine whether there is a preset object in the gesture circle device image; Step S36: Determine the preset object within the gesture circle device image as the processed device image selected by the user.

[0064] Specifically, the preset object includes the image of the processing device. After determining the range of the gesture circle device image in the device scene image, it can be determined whether there is an image of the processing device in the gesture circle device image, so as to determine the image of the processing device that falls within the range of the gesture circle device image as the image of the processing device selected by the user.

[0065] After step S36, it further includes: Step S361: Determine whether the number of the processing device images selected by the user meets a preset number; Step S362: If the number of the processing device images selected by the user meets the preset number, execute step S4.

[0066] Specifically, the processing device images selected by the user can be one or more. That is, the display module 14 can display the operation parameter interface 141 of one processing device, or can also display the operation parameter interfaces 141 of multiple processing devices at the same time. The preset number is the number of processing devices for which the operation parameter interface 141 needs to be displayed as preset. The processor 15 is used to determine whether the number of the processing device images within the range of the gesture circle device image matches the preset number. If the number of the processing device images meets the preset number, the subsequent steps are continued.

[0067] After step S362, it further includes step S363: If the number of the processing device images selected by the user does not meet the preset number, switch the color of the reference benchmark circle in the spatial interface to form a prompt identifier.

[0068] Specifically, please refer to Figure 7 , when the processor 15 determines that the number of the processing device images selected by the user does not match the preset number, the color of the reference benchmark circle 142 can be changed, thereby forming a prompt identifier 144. For example, the prompt identifier 144 can be red. That is, when the reference benchmark circle 142 changes from white to red, it is the prompt identifier 144, indicating that the number of the processing device images selected by the user does not match the preset number.

[0069] In some embodiments, when executing step S3, the following steps can also be included: Step S311: Based on the eye state information and the virtual reality scene, determine the spatial position of the user's eyes; Step S312: Based on the selection gesture image and the virtual reality scene, determine the spatial position of the device selection gesture; Step S313: Based on the virtual reality scene, connect the spatial position of the user's eyes and the spatial position of the device selection gesture in series to form a spatial straight line; Step S314: Based on the virtual reality scene, identify whether there is a virtual processing device on the spatial straight line; Step S315: Determine the virtual processing device on the spatial straight line as the processing device image selected by the user.

[0070] Specifically, steps S311 to S315 are executed based on a virtual reality scenario. By determining the spatial position of the eye E and the spatial position of the device selection gesture in the virtual reality scenario, and connecting the spatial position of the eye E and the spatial position of the device selection gesture in the virtual reality scenario to form a spatial straight line, it is possible to directly determine whether there is a virtual processing device on the straight line, thereby determining the processed device image selected by the user. Steps S31 to S36 and steps S311 to S315 can be carried out simultaneously, thereby enhancing the recognition accuracy of the processed device image selected by the user.

[0071] Step S4: Based on the processed device image selected by the user, display a selection success mark, and determine one of the processed device images selected by the user as the actual processing device to be selected; Specifically, please refer to Figure 8 , the display of the selection success mark 143 is specifically: in the spatial interface, switch the color of the reference datum circle to form the success mark. For example, the selection success mark 143 can be blue, that is, when the color of the reference datum circle 142 changes from white to blue, it is the selection success mark 143, indicating that the user has successfully selected the selected processed device image.

[0072] Step S4 further includes: Step S41: Obtain the parameter retrieval gesture image of the user; Step S42: Determine whether the parameter retrieval gesture image has the gesture circle; Step S43: If the parameter retrieval gesture image has the gesture circle, determine whether there is a finger gesture extending on the gesture circle in the parameter retrieval gesture image; Step S44: If there is a finger gesture extending on the gesture circle in the parameter retrieval gesture image, determine whether the finger gesture in the parameter retrieval gesture image matches a preset gesture; Step S45: If the finger gesture in the parameter retrieval gesture image matches the preset gesture, execute step S5.

[0073] Specifically, after successfully identifying the processed device selected by the user, the user is also required to confirm to execute subsequent parameter retrieval. In some embodiments, the user needs to make a parameter retrieval gesture, and after the camera module 12 acquires and identifies the parameter retrieval gesture of the user, the subsequent steps are executed.

[0074] In some embodiments, the parameter retrieval gesture is specifically that the thumb and index finger form a gesture circle H, and the middle finger, ring finger, and little finger extend outward to form a finger gesture H2. Multiple images of the parameter retrieval gesture are pre-stored in the processor 15. When the camera module 12 captures the parameter retrieval gesture image, through the deep learning algorithm, the parameter retrieval gesture image captured by the camera module 12 can be compared with the parameter retrieval gesture images pre-stored in the processor 15, so as to determine whether the parameter retrieval gesture image matches.

[0075] In some embodiments, after step S45, it further includes: Step S46: Obtain the voice signal of the user; Step S47: Identify the confirmation parameter retrieval voice in the voice signal; Step S48: Based on the confirmation parameter retrieval voice, execute step S5 (which will be described below).

[0076] Specifically, steps S46 to S48 are the voice confirmation link. That is, the user needs to make the parameter retrieval gesture and issue the confirmation voice in sequence to determine the selected processing device and ensure the accuracy of the selection.

[0077] Step S5: Obtain the operating parameters of the selected actual processing device and display the operating parameter interface of the selected actual processing device.

[0078] Before executing step S5, it further includes: Step S51: Based on the selected actual processing device, establish a communication connection with the actual processing device.

[0079] Specifically, in some embodiments, the head-mounted intelligent device 10 is in an unconnected state with the processing device. The head-mounted intelligent device 10 stores the identification code of each processing device. When the selected actual processing device is determined, the head-mounted intelligent device 10 can establish a communication connection with the actually selected processing device through the identification code of the selected processing device, so as to obtain the operating parameters of the selected processing device.

[0080] In other embodiments, the head-mounted intelligent device 10 can also maintain a connection relationship with all processing devices all the time and directly obtain the operating parameters of the selected processing device after determining the selected processing device.

[0081] In some embodiments, after executing step S5, it further includes: Step S61: Obtain the manipulation action image of the user based on the operating parameter interface; Step S62: Correspondingly control the selected actual processing device according to the manipulation action image.

[0082] Specifically, after obtaining the operating parameters of the processing equipment, the non-contact processing equipment interaction method further includes non-contact control of the processing equipment. That is, the user can make a manipulation action. After the camera module 12 of the head-mounted intelligent device 10 captures the manipulation action, it can send the manipulation action image to the processor 15. After the processor 15 recognizes the manipulation action image, it generates a corresponding control instruction and sends it to the selected processing equipment, thereby controlling the selected processing equipment.

[0083] In some embodiments, the non-contact processing equipment interaction method further includes: Step 71: Based on the virtual reality scene, display the operating parameter interface of the selected actual processing equipment on one side of the virtual processing equipment; or, Step 72: Based on the virtual reality scene, hide the virtual processing equipment and display the operating parameter interface of the actual processing equipment on one side of the actual processing equipment.

[0084] Specifically, the non-contact processing equipment interaction method further includes displaying the operating parameter interface of the selected actual processing equipment on the server 30, so that the user using the remote terminal and the user located in the actual processing equipment scene can view the operating parameters of the selected processing equipment at the same time. The operating parameter interface can be displayed on one side of the virtual processing equipment, can directly replace the virtual processing equipment, or can be superimposed on the virtual processing equipment. This application does not limit this.

[0085] The non-contact processing equipment interaction method provided by the embodiments of the present application can obtain the scene image of the actual processing equipment in front of the user's eyes by acquiring the scene of the actual processing equipment in front of the user's eyes. By acquiring the eye state information of the user and the image of the user's selection gesture, combined with the scene image, the processing equipment selected by the user can be identified, and the operating parameter interface of the processing equipment selected by the user can be displayed to the user, so as to facilitate the user to obtain the operating parameter information of the selected processing equipment, realize the interaction with the processing equipment in a non-contact state, and is beneficial to quickly and conveniently determine the processing equipment selected by the user when facing multiple processing equipment, thereby improving the non-contact interaction experience.

[0086] The embodiments of the present application further provide a storage medium for storing a computer program or code. When the computer program or code is executed by a processor, the non-contact processing equipment interaction method of the embodiments of the present application is implemented.

[0087] A storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The storage medium includes, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassette, tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0088] Those of ordinary skill in the art should recognize that the above embodiments are only used to illustrate the present application and are not intended to limit the present application. As long as appropriate changes and variations are made within the scope of the spirit of the present application, they fall within the scope of protection required by the present application.

Claims

1. A non-contact processing equipment interaction method, applied to a head-mounted intelligent device, characterized in that: include: Acquire a device scene image based on a scene of an actual processing device in front of a user, wherein the device scene image includes a processing device image and a selection gesture image of the user formed in the processing device image; Acquiring eye status information of the user; Identifying the processing equipment image selected by the user according to the eye state information and the selection gesture image; Based on the processing equipment image selected by the user, displaying a selection success mark, and determining the processing equipment image selected by the user as the actual processing equipment selected; The operating parameters of the selected actual processing equipment are obtained, and an operating parameter interface of the selected actual processing equipment is displayed.

2. The non-contact processing equipment interaction method according to claim 1, characterized in that: The selected gesture image has a gesture circle, and the step of identifying the processing equipment image selected by the user according to the eye state information and the selected gesture image comprises: Display reference datum circle in the space interface; Based on the eye state information, the reference reference circle, and the gesture circle, determining the pupil center, the reference reference center, and the gesture center in a one-to-one correspondence; Determining whether the pupil center, the reference base center, and the gesture center are on a preset condition straight line; If the pupil center, the reference base center and the gesture center are in a preset straight line, the device scene image within the gesture circle is determined as a gesture circle device image; Determine whether there is a preset object in the gesture circle device image; The preset object in the gesture circle device image is determined as the processing device image selected by the user.

3. The non-contact processing equipment interaction method according to claim 2, characterized in that: The step of displaying a successful selection mark comprises: In the space interface, the color of the reference base circle is switched to form the selection success mark.

4. The non-contact processing equipment interaction method according to claim 2, characterized in that: Also includes: Obtaining the user's parameters to retrieve gesture images; Determine whether the parameter-retrieving gesture image has the gesture circle; If the parameter-retrieving gesture image has the gesture circle, determining whether a finger gesture extends on the gesture circle in the parameter-retrieving gesture image; If the finger gesture is extended on the gesture circle in the parameter-retrieving gesture image, determining whether the finger gesture in the parameter-retrieving gesture image matches a preset gesture; If the finger gesture in the parameter-retrieving gesture image matches the preset gesture, the step of "obtaining the operating parameters of the selected actual processing equipment" is executed.

5. The non-contact processing equipment interaction method according to claim 1, characterized in that: Also includes: Determining whether the number of the processing equipment images selected by the user meets a preset number; If the number of the processing equipment images selected by the user meets the preset number, execute the step of "displaying a selection success mark based on the processing equipment images selected by the user, and determining the processing equipment images selected by the user as the actual processing equipment selected".

6. The non-contact processing equipment interaction method according to claim 1, characterized in that: Also includes: Acquiring a voice signal of the user; Recognize the confirmation parameter in the sound signal to retrieve the voice; Based on the voice confirmation parameter, the step of "obtaining the operating parameters of the actual processing equipment selected" is executed.

7. The non-contact processing equipment interaction method according to claim 2, characterized in that: Also includes: If the user selects multiple processing equipment, in the space interface, the color of the reference base circle is switched to form a prompt mark.

8. The non-contact processing equipment interaction method according to claim 1, characterized in that: Also includes: Acquiring a manipulation action image of the user based on the operation parameter interface; According to the manipulation action image, the selected actual processing equipment is controlled accordingly.

9. The non-contact processing equipment interaction method according to claim 1, characterized in that: Also includes: Scanning a scene of the actual processing equipment to form the equipment scene image, wherein the actual processing equipment has an identification diagram; Based on the equipment scene image, construct a virtual reality scene with virtual processing equipment; Based on the identification image of each of the actual processing equipment in the equipment scene image, identifying the identification code of each of the actual processing equipment; In the virtual reality scene, the identification code of each of the actual processing equipment is calibrated to each of the virtual processing equipment in a one-to-one correspondence, so that the operating parameters of the actual processing equipment can be associated with the virtual processing equipment based on the identification code.

10. The non-contact processing equipment interaction method according to claim 1, characterized in that: Also includes: Scanning the scene of the actual processing equipment to form the equipment scene image; Based on the equipment scene image, construct a virtual reality scene with virtual processing equipment; Sending the virtual reality scene to a server; Receive the calibrated virtual reality scene fed back by the server, wherein each of the virtual processing devices in the calibrated virtual reality scene is calibrated with an identification code, and the identification code calibrated for each of the virtual processing devices corresponds one-to-one with the identification code of each of the actual processing devices, so as to associate the operating parameters of the actual processing devices with the virtual processing devices based on the identification codes.

11. The non-contact processing equipment interaction method according to claim 9 or 10, characterized in that: The step of identifying the processing equipment image selected by the user according to the eye state information and the selection gesture image comprises: Determining a spatial position of the user's eyes based on the eye state information and the virtual reality scene; Determining a spatial position of the device selection gesture based on the selection gesture image and the virtual reality scene; Based on the virtual reality scene, the spatial position of the user's eyes and the spatial position of the device selection gesture are connected in series to form a spatial straight line; Based on the virtual reality scene, identifying whether there is the virtual processing equipment on the straight line in the space; The virtual processing equipment on the spatial straight line is determined as the processing equipment image selected by the user.

12. The non-contact processing equipment interaction method according to claim 11, characterized in that: Also includes: Based on the selected actual processing equipment, a communication connection is established with the actual processing equipment.

13. The non-contact processing equipment interaction method according to claim 9 or 10, characterized in that: Also includes: Based on the virtual reality scene, displaying an operating parameter interface of the selected actual processing equipment on one side of the virtual processing equipment; or, Based on the virtual reality scene, the virtual processing equipment is hidden, and the operating parameter interface of the actual processing equipment is displayed on the side of the actual processing equipment.

14. A storage medium for storing a computer program or code, characterized in that: When the computer program or code is executed by a processor, the non-contact processing equipment interaction method as described in any one of claims 1 to 13 is implemented.

15. A head-mounted intelligent device, characterized in that: include: A camera module, used for acquiring a device scene image based on a scene of an actual processing device in front of a user, wherein the device scene image includes a processing device image and a selection gesture image of the user formed in the processing device image; An eye tracking module, used to obtain eye status information of the user; A processor, configured to identify the processing equipment image selected by the user according to the eye state information and the selection gesture image; A display module, used for displaying a selection success mark based on the processing equipment image selected by the user; The processor is further used to determine one of the processing equipment images selected by the user as the selected actual processing equipment; and to obtain the operating parameters of the selected actual processing equipment; The display module is also used to display the operating parameter interface of the selected actual processing equipment.

16. A non-contact processing equipment interactive system, characterized in that: include: Actual processing equipment; Head-mounted smart device for: Acquire a device scene image formed based on a scene of the actual processing device in front of the user, wherein the device scene image includes a processing device image and a selection gesture image of the user formed in the processing device image; Acquiring eye status information of the user; Identifying the processing equipment image selected by the user according to the eye state information and the selection gesture image; Based on the processing equipment image selected by the user, displaying a selection success mark, and determining the processing equipment image selected by the user as the actual processing equipment selected; Acquire the operating parameters of the selected actual processing equipment, and display the operating parameter interface of the selected actual processing equipment; A server is used to construct a virtual reality scene based on the scene of the actual processing equipment, each virtual processing equipment in the virtual reality scene corresponds to each actual processing equipment, and the server is used to associate the operating parameters of the selected actual processing equipment with the corresponding virtual processing equipment, so that the corresponding virtual processing equipment is associated with the operating parameters; The remote terminal is used to obtain the virtual reality scene and display the corresponding virtual processing equipment and the associated operating parameters.

Images