Voice positioning method and device based on mixed reality equipment, equipment and medium

Abstract

The invention discloses a voice positioning method and device based on mixed reality equipment, equipment and a medium. The method is characterized by comprising the following steps: in response to a voice control instruction input by a user for a user interaction image, determining a target coordinate corresponding to the voice control instruction; determining a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinate; and marking the target image object in the user interaction image through a preset marking mode, and displaying the positioning coordinate information. According to the invention, the object positioned by the user through voice can be marked in the mixed reality picture, and the real coordinate of the object is displayed, so that the function of constructing interactive feedback among the virtual world, the real world and the user through voice is realized, and the efficiency of acquiring positioning information by the user is improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular, to a voice positioning method, device, equipment, and medium based on a mixed reality device. Background Art

[0002] A mixed reality device is a device that combines virtual images and the real world. The mixed display device can provide a "virtual-in-real" semi-immersive experience for users through technologies such as spatial positioning, holographic projection, human-computer interaction, and sensing. When a user uses a mixed reality device, the user can see and interact with virtual object images in the real environment, and as the user's operations and the real environment change, the positions and states of the virtual object images also change. The operations of the mixed reality device can be to select an object by setting an infrared camera to monitor the changes in the user's pupils and eye behaviors, or to implement control logic by recognizing the user's gestures, or to recognize the user's voice through a built-in microphone to implement operations on the objects in the picture. However, the current operation methods only select objects and display the relevant information of the objects, and cannot locate the actual positions of real objects. Summary of the Invention

[0003] The present invention provides a voice positioning method, device, equipment, and medium based on a mixed reality device to solve the technical problem in the prior art that the actual position of an object cannot be located by voice in a mixed reality device.

[0004] According to one aspect of the present invention, there is provided a voice positioning method based on a mixed reality device, including:

[0005] Responding to a voice control instruction for a user interaction image input by a user to determine a target coordinate corresponding to the voice control instruction;

[0006] Determining a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinate;

[0007] Marking the target image object in the user interaction image by a preset marking method and displaying the positioning coordinate information.

[0008] According to another aspect of the present invention, there is provided a voice positioning device based on a mixed reality device, including:

[0009] A voice module, configured to respond to a voice control instruction for a user interaction image input by a user to determine a target coordinate corresponding to the voice control instruction;

[0010] A positioning module, configured to determine a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinate;

[0011] An object marking module, configured to mark the target image object in the user interaction image by a preset marking method and display the positioning coordinate information.

[0012] According to another aspect of the present invention, there is provided a mixed reality device, the mixed reality device comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the voice positioning method based on the mixed reality device according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for implementing the voice positioning method based on the mixed reality device according to any embodiment of the present invention when executed by a processor.

[0017] The technical solution of the embodiment of the present invention, by responding to a voice control instruction of a user interaction image for a user input, determines a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinates; marks the target image object in the user interaction image by a preset marking method and displays the positioning coordinate information. By responding to the voice of the user interaction image, it can identify the real object corresponding to the user interaction image, and then mark the real object in the user interaction image and display the actual positioning information of the real object, and can directly identify and locate the real object by voice, solving the technical problem in the prior art that the real position of the object cannot be located by voice in the mixed reality device, realizing the real-time positioning of the real object in the mixed reality image, improving the efficiency of the user to obtain the positioning information of the real object, effectively reducing the time of user decision-making, and improving the accuracy and efficiency of user decision-making.

[0018] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0020] Figure 1 The flowchart of a voice positioning method based on a mixed reality device provided by an embodiment of the present invention;

[0021] Figure 2 The flowchart of another voice positioning method based on a mixed reality device provided by an embodiment of the present invention;

[0022] Figure 3 The flowchart of another voice positioning method based on a mixed reality device provided by an embodiment of the present invention;

[0023] Figure 4 The structural schematic diagram of a voice positioning device based on a mixed reality device provided by an embodiment of the present invention;

[0024] Figure 5 The structural schematic diagram of the mixed reality device 10 that can be used to implement the embodiments of the present invention is shown. Detailed implementation manners

[0025] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above accompanying drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0027] Figure 1The present invention provides a flowchart of a voice positioning method based on a mixed reality device. This embodiment is applicable to the situation where a user uses a mixed reality device to position an object in a mixed reality image displayed by the mixed reality device. This method can be executed by a voice positioning device based on the mixed reality device. The voice positioning device based on the mixed reality device can be implemented in the form of hardware and / or software, and the voice positioning device based on the mixed reality device can be configured in the mixed reality device. As Figure 1 shown, the method includes:

[0028] S110. In response to a voice control instruction input by the user for the user interaction image, determine a target coordinate corresponding to the voice control instruction.

[0029] Among them, the user interaction image can be an image displayed by the mixed reality device to the user. The user interaction image displayed by the mixed reality device is obtained by combining a real-time reality image captured by the mixed reality device and virtual elements generated by the mixed reality device. The user can see both the real reality image and the virtual image in the user interaction image. It should be noted that the virtual elements of the user interaction image at least include a virtual two-dimensional coordinate system, and the user can see the two-dimensional coordinate system and the coordinate points of the two-dimensional coordinate system in the user interaction image.

[0030] Optionally, in the present invention, the mixed reality device generates virtual elements corresponding to the two-dimensional coordinate system on the basis of the real-time reality image, and combines the real-time reality image and the virtual elements corresponding to the two-dimensional coordinate system to obtain the user interaction image.

[0031] Among them, the voice control instruction can be voice instruction information for the user to control the mixed reality device. It should be noted that a microphone is set in the mixed reality device, and the voice of the user can be captured through the microphone, and then the voice control instruction can be obtained.

[0032] Among them, the target coordinate can be the coordinate selected by the user's voice control instruction in the user interaction image. It should be noted that since there is a virtual two-dimensional coordinate system in the user interaction image, when the user uses the mixed reality device, the user issues a voice control instruction to the mixed reality device by selecting the coordinates of the virtual two-dimensional coordinate system based on the user interaction image displayed by the mixed reality device, and the mixed reality device obtains the target coordinate through the voice control instruction. Exemplarily, the user selects the coordinate (2, 2) in the two-dimensional coordinate system of the user interaction image and issues a voice control instruction to the mixed reality device, and the mixed reality device recognizes the voice control instruction and obtains the target coordinate (2, 2).

[0033] Specifically, when a user uses a mixed reality device, based on the mixed reality device displaying a user interaction image, the user inputs a voice control instruction to the mixed reality device, and the mixed reality device determines the target coordinates corresponding to the voice control instruction in response to the voice control instruction input by the user for the user interaction image.

[0034] Optionally, in another alternative embodiment of the present invention, before determining the target coordinates corresponding to the voice control instruction in response to the voice control instruction input by the user for the user interaction image, it further includes:

[0035] Receiving a control voice input by the user; parsing the control voice through a pre-trained voice recognition model to determine the voice control instruction.

[0036] Among them, the control voice can be the voice output by the user to the mixed reality device through the microphone of the mixed reality device. It should be noted that the user selects coordinates based on the user interaction image displayed by the mixed reality device and inputs the control voice to the mixed reality device.

[0037] Among them, the voice recognition model can be a neural network model pre-trained for voice recognition. It should be noted that the voice recognition model can map the acoustic signal corresponding to the control voice to a text sequence. The mixed reality device converts the acoustic signal corresponding to the control voice into a data signal, extracts the acoustic features of the data signal, performs text inference on the acoustic features through the voice recognition model to obtain the text sequence corresponding to the control voice, and after obtaining the text sequence, performs natural language understanding on the text sequence to convert it into a voice control instruction.

[0038] Specifically, receive the control voice input by the user, parse the control voice through a pre-trained voice recognition model, and determine the voice control instruction.

[0039] Optionally, in another alternative embodiment of the present invention, before determining the target coordinates corresponding to the voice control instruction in response to the voice control instruction input by the user for the user interaction image, it further includes:

[0040] Capturing a real-time reality image, generating a corresponding mixed reality image according to the real-time reality image; dividing a two-dimensional coordinate system in the mixed reality image through a pre-set virtual two-dimensional coordinate system generation method to determine the user interaction image.

[0041] Among them, the real-time reality image can be a real image captured in real time by a camera set on the mixed reality device. It should be noted that various different types of cameras and sensors are usually set in the mixed reality device for real-time capturing of the real-world environment, perceiving the depth of the real-world environment, detecting the inertia of the mixed reality device, and capturing the dynamics and voices of users. Exemplarily, an RGB perspective camera, a ToF (Time of Flight) sensor, an inertial sensor, an infrared camera, a microphone, etc. are set.

[0042] Optionally, when the user uses the mixed reality device, the mixed reality device will capture the dynamics and / or voices of the user's eyes in real time, and adjust the shooting direction of the camera based on the captured dynamics and / or voices of the user's eyes, that is, control the shooting direction of the camera to meet the dynamics and / or voices of the user's eyes, and capture the image of the real-world environment in real time to obtain a real-time reality image.

[0043] Among them, the mixed reality image can be an image obtained by the mixed reality device by combining virtual elements with the real-time reality image. It should be noted that the user interaction image is a mixed reality image in which the virtual elements at least include a two-dimensional coordinate system.

[0044] Among them, the virtual two-dimensional coordinate system generation method can be a method preset for generating a virtual two-dimensional coordinate system for the mixed display image. Optionally, when the virtual two-dimensional coordinate system generation method divides the virtual two-dimensional coordinate system for the mixed reality image, it first selects the origin of the virtual two-dimensional coordinate system. The user can be set as the origin of the two-dimensional coordinate system, and the X / Y axes are set to be aligned with the orientation of the mixed reality device and the ground of the real world. The ground plane of the real-time reality environment is detected, and the coordinate axes and coordinate points of the virtual two-dimensional coordinate system are rendered in the mixed reality image to form a virtual two-dimensional coordinate system.

[0045] Optionally, after the virtual two-dimensional coordinate system is displayed in the mixed reality image to obtain the user interaction image, due to the movement of the user or the movement of the mixed reality device by the user, the position of the mixed reality device changes, and then the origin of the coordinate system changes. The mixed reality device will choose to update the origin of the virtual two-dimensional coordinate system based on the user or the mixed reality device, and re-render the virtual two-dimensional coordinate system to achieve real-time display of an accurate virtual two-dimensional coordinate system. Among them, when the mixed reality device moves, the inertial sensor is selected to detect the movement and the amount of movement of the mixed reality device.

[0046] Optionally, in another embodiment of the present invention, after obtaining the user interaction image, when the mixed reality device moves, the position of the mixed reality device changes, which in turn causes the origin of the coordinate system to change. The mixed reality device remembers the origin of the virtual two-dimensional coordinate system and does not update the origin of the virtual coordinate system. According to the position of the mixed reality device after movement in the virtual two-dimensional coordinate system, the coordinate axes and coordinate points of the virtual coordinate system in the mixed reality image are updated to achieve real-time rendering of the virtual two-dimensional coordinate system.

[0047] Specifically, capture the real-time reality image, generate the corresponding mixed reality image according to the real-time reality image, divide the two-dimensional coordinate system in the mixed reality image through a preset virtual two-dimensional coordinate system generation method, and determine the user interaction image.

[0048] S120. Determine the target image object in the user interaction image and the positioning coordinate information corresponding to the target image object according to the target coordinate.

[0049] Among them, the target image object may be the object corresponding to the target coordinate in the user interaction image. It should be noted that the target image object is an object in the real real environment. The user can see the virtual two-dimensional coordinate system in the user interaction image. The mixed reality device identifies the coordinate point corresponding to the target coordinate of the virtual two-dimensional coordinate system and uses the object at the coordinate point as the target image object.

[0050] Optionally, in the present invention, the user interaction image displays the real real environment and the virtual elements generated by the mixed reality device. The virtual elements generated by the mixed reality device at least include a virtual two-dimensional coordinate system. When the mixed reality device obtains the target coordinate, it can identify the target image object of the target coordinate in the real real environment.

[0051] Among them, the positioning coordinate information may be the coordinate information of the target image object in the real environment. It should be noted that the target image object is a real object in the real environment, and the positioning coordinate information of the target image object is the coordinate information of the object in the real environment. Exemplarily, the positioning coordinate information may be longitude and latitude coordinates.

[0052] Optionally, the mixed reality device identifies the target image object corresponding to the target coordinate and obtains the positioning coordinate information of the target image object.

[0053] Specifically, determine the target image object in the user interaction image and the positioning coordinate information corresponding to the target image object according to the target coordinate.

[0054] S130. Mark the target image object in the user interaction image through a preset marking method and display the positioning coordinate information.

[0055] Among them, the preset marking method can be a method preset for marking target objects in the user interaction image. It should be noted that the preset marking method generates virtual elements through a reality device to mark real target image objects during marking. This virtual element can highlight the target image object in the user interaction image, and the virtual element can be called a marking virtual element. Exemplarily, the marking virtual element can enclose the target image object with a virtual circle, and the color of the virtual circle can be set to any color, such as red.

[0056] Optionally, after marking the target image object, process the marking of the target image object. The mixed reality device constructs a virtual display element for displaying positioning coordinate information and displays the virtual display element in the user interaction image to display the positioning coordinate information.

[0057] Specifically, mark the target image object in the user interaction image through the preset marking method and display the positioning coordinate information corresponding to the target image object.

[0058] The technical solution of the embodiment of the present invention, by responding to the voice control instruction of the user interaction image for user input, determines the target image object in the user interaction image and the positioning coordinate information corresponding to the target image object according to the target coordinate; marks the target image object in the user interaction image through the preset marking method and displays the positioning coordinate information. By responding to the voice of the user interaction image, it can identify the real object corresponding to the user interaction image, and then mark the real object in the user interaction image and display the actual positioning information of the real object. It can directly identify and locate the real object by voice, solves the technical problem in the prior art that the real position of the object cannot be located by voice in the mixed reality device, realizes the real-time positioning of the real object in the mixed reality image, improves the efficiency of the user to obtain the positioning information of the real object, effectively reduces the time for user decision-making, and improves the accuracy and efficiency of user decision-making.

[0059] Figure 2 It is a flowchart of another voice positioning method based on a mixed reality device provided by an embodiment of the present invention. The relationship between this embodiment and the above embodiment is the specific method for the mixed reality device to obtain the positioning coordinate information of the target image object. As Figure 2 shown, the method includes:

[0060] S210. Respond to the voice control instruction of the user interaction image for user input and determine the target coordinate corresponding to the voice control instruction.

[0061] S220. Scan the image area corresponding to the target coordinate in the user interaction image and determine the target image object in the image area.

[0062] Optionally, after the mixed reality device obtains the target coordinates, it scans the image area corresponding to the target coordinates in the displayed user interaction image, scans the object corresponding to the image area, and uses the scanned object as the target image object.

[0063] Specifically, scan the image area corresponding to the target coordinates in the user interaction image to determine the target image object in the image area.

[0064] S230. Perform coordinate positioning based on the target image object and the target coordinates to determine the positioning coordinate information.

[0065] Specifically, the mixed reality device performs coordinate positioning based on the target image object and the target coordinates to determine the positioning coordinate information.

[0066] Optionally, in another optional embodiment of the present invention, the performing coordinate positioning based on the target image object and the target coordinates to determine the positioning coordinate information includes:

[0067] Perform image recognition on the target image object through a pre-trained large image recognition model to identify the target object information corresponding to the target image object; perform coordinate positioning based on the target object information and the target coordinates to determine the positioning coordinate information.

[0068] Among them, the large image recognition data model can be a pre-trained image processing model for object recognition of objects in mixed reality images; the large image recognition data model can identify the feature information of objects in mixed reality images. It should be noted that the large image recognition data model can be pre-trained and deployed on the server side of the mixed reality device or in the mixed reality device. After the mixed reality device generates a mixed reality image, the large image recognition data is called to perform real-time image scanning on the mixed reality image to identify the target object feature information in the mixed reality image. Exemplarily, the large image recognition data model can be constructed using a convolutional neural network and a multi-modal Transformer model to obtain a preliminary model. By collecting a large number of historical mixed reality images of the mixed reality device as training data, the preliminary model is trained with the training data, and the model is optimized for different military scenarios. When the accuracy rate of the preliminary model reaches 99.9%, the preliminary model is used as the large image recognition data model.

[0069] Among them, the target object information can be the object information of the target image object. Exemplarily, the target image object can be a tank, and the target object information can be the model of the tank; the target image object can be a house, and the target object information can be the style of the house.

[0070] Specifically, an image recognition large model that has been pre-trained is used to perform image recognition on the target image object to identify the target object information corresponding to the target image object; coordinate positioning is performed based on the target object information and the target coordinates to determine the positioning coordinate information.

[0071] Optionally, in another optional embodiment of the present invention, the performing coordinate positioning according to the target object information and the target coordinates to determine the positioning coordinate information includes:

[0072] Combining the target object information and the target coordinates into object retrieval information; sending the object retrieval information to the mixed reality cloud for positioning search to determine the positioning coordinate information corresponding to the target image object.

[0073] Among them, the object retrieval information can be used to retrieve the positioning coordinate information and the object display information of the target image object. It should be noted that the object retrieval information is combined and integrated from the target object information and the target coordinates into object retrieval information.

[0074] Optionally, the object retrieval information is integrated into a request message, and the request message is sent to the mixed reality cloud to implement the positioning search for the target image object.

[0075] Among them, the mixed reality cloud can be a cloud data center that is pre-set to be able to identify the position of the target image object.

[0076] Optionally, after the mixed reality cloud receives the data retrieval information, it determines the positioning coordinate information of the target image object based on the data retrieval information, and sends the positioning coordinate information of the target image object to the mixed reality device.

[0077] Optionally, the mixed reality device combines the target object information and the target coordinates into object retrieval information, and sends the object retrieval information to the mixed reality cloud for fast positioning search; the mixed reality cloud performs positioning according to the object retrieval information to determine the positioning coordinate information corresponding to the target image object.

[0078] Optionally, in the present invention, when the mixed reality cloud receives the object retrieval information, it identifies the positioning coordinates of the mixed reality device, performs positioning calculations based on the positioning coordinates of the mixed reality device and the target coordinates, identifies the calculated positioning coordinates of the target image object, and calls the positioning device to scan the calculated positioning coordinates to identify whether there is an object corresponding to the target object information at the calculated positioning coordinates. If there is, it is considered that the calculated positioning coordinates are the positioning coordinate information of the target image object, and the positioning coordinate information is replied to the mixed display device; if not, a failure message is replied to the mixed reality device. Exemplarily, the failure message can be "unable to locate", and the positioning device can be a radar and a satellite.

[0079] S240. Mark the target image object in the user interaction image through a preset marking method, and display the positioning coordinate information.

[0080] Optionally, in another alternative embodiment of the present invention, after combining the target object information and the target coordinates into object retrieval information, it further includes:

[0081] Send the object retrieval information to the mixed reality cloud for information search, determine the object display information corresponding to the target object information, and display the object display information in the user interaction image.

[0082] Among them, the mixed reality cloud can also store at least one object display information; the object display information can be related information introducing the target image object. Exemplarily, taking the target image object as a bank, the object display information can be the name of the bank and the creation time, etc.

[0083] Optionally, the object display information of each object is pre-stored in the mixed reality cloud. If there is no object display information corresponding to the target image object in the mixed reality cloud, the mixed reality cloud connects to the Internet to search for the target image object to obtain the object display information corresponding to the target object information.

[0084] Optionally, the mixed reality device constructs a virtual display element for displaying the object display information, and displays the virtual display element in the user interaction image to display the object display information.

[0085] Specifically, send the object retrieval information to the mixed reality cloud for information search, determine the object display information corresponding to the target object information, and display the object display information in the user interaction image.

[0086] In the technical solution of the embodiment of the present invention, the mixed reality device identifies the target image object to obtain relevant object information, then locates the actual position coordinates of the target image object, and displays the actual position coordinates in the image of the mixed reality device, and can also display the object display information of the target image object. Through the positioning of the target image object by the mixed reality device, the technical problem in the prior art that the actual position of the object cannot be located by voice in the mixed reality device is solved, the real-time positioning of the real object in the mixed reality image is realized, the efficiency of the user to obtain the positioning information of the real object is improved, the time for the user to make a decision is effectively reduced, and the accuracy and efficiency of the user's decision-making are improved.

[0087] Figure 3 It is a flowchart of another voice positioning method based on a mixed reality device provided by the embodiment of the present invention, as Figure 3 shown:

[0088] S310. Capture real-time reality images.

[0089] Specifically, the user captures a real image through the camera device of the mixed reality device.

[0090] S320. Generate a user interaction image based on the real-time real image.

[0091] Specifically, the mixed reality device generates virtual elements corresponding to a two-dimensional coordinate system on the basis of the real-time real image, and combines the real-time real image and the virtual elements corresponding to the two-dimensional coordinate system to obtain a user interaction image.

[0092] S330. Receive the control voice input by the user.

[0093] S340. Process the control voice.

[0094] Specifically, the control voice is parsed through a pre-trained speech recognition model to determine the voice control instruction.

[0095] S350. In response to the voice control instruction, identify the target image object in the user interaction image based on the target coordinates corresponding to the voice control instruction, and mark the target image object in the user interaction image.

[0096] Specifically, in response to the voice control instruction input by the user for the user interaction image, determine the target coordinates corresponding to the voice control instruction, and mark the target image object in the user interaction image by a preset marking method.

[0097] S360. Perform coordinate positioning according to the target image object and the target coordinates to determine the positioning coordinate information.

[0098] S370. Display the positioning coordinate information of the target image object.

[0099] The technical solution of the embodiment of the present invention solves the technical problem in the prior art that the real position of an object cannot be located by voice in a mixed reality device through the positioning of the target image object by the mixed reality device, realizes the real-time positioning of the real object in the mixed reality image, improves the efficiency of the user to obtain the positioning information of the real object, effectively reduces the time for the user to make a decision, and improves the accuracy and efficiency of the user's decision-making.

[0100] Figure 4 It is a schematic structural diagram of a voice positioning device based on a mixed reality device provided by an embodiment of the present invention. As Figure 4 shown, the device includes: a voice module 410, a positioning module 420, and an object marking module 430; wherein,

[0101] The voice module 410 is configured to determine the target coordinates corresponding to the voice control instruction in response to the voice control instruction input by the user for the user interaction image;

[0102] A positioning module 420, configured to determine a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinates;

[0103] An object marking module 430, configured to mark the target image object in the user interaction image by a preset marking method and display the positioning coordinate information.

[0104] The technical solution of the embodiment of the present invention, by responding to a voice control instruction of a user interaction image input by a user, determines a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinates; marks the target image object in the user interaction image by a preset marking method and displays the positioning coordinate information. By responding to the voice of the user interaction image, it can identify the real object corresponding to the user interaction image, and then mark the real object in the user interaction image and display the actual positioning information of the real object, and can directly locate the real object through voice recognition, solving the technical problem in the prior art that the real position of an object cannot be located by voice in a mixed reality device, realizing the real-time positioning of the real object in the mixed reality image, improving the efficiency of the user to obtain the positioning information of the real object, effectively reducing the time of user decision-making, and improving the accuracy and efficiency of user decision-making.

[0105] Optionally, the positioning module is specifically configured to:

[0106] Scan an image area corresponding to the target coordinates in the user interaction image to determine a target image object in the image area;

[0107] Perform coordinate positioning according to the target image object and the target coordinates to determine the positioning coordinate information.

[0108] Optionally, the positioning module is specifically further configured to:

[0109] Perform image recognition on the target image object through a pre-trained large image recognition model to recognize target object information corresponding to the target image object;

[0110] Perform coordinate positioning according to the target object information and the target coordinates to determine the positioning coordinate information.

[0111] Optionally, the positioning module is specifically further configured to:

[0112] Combine the target object information and the target coordinates into object retrieval information;

[0113] Send the object retrieval information to the mixed reality cloud for location search to determine the location coordinate information corresponding to the target image object.

[0114] Optionally, the device further includes an information display module;

[0115] The information display module is configured to send the object retrieval information to the mixed reality cloud for information search, determine the object display information corresponding to the target object information, and display the object display information in the user interaction image.

[0116] Optionally, the device further includes a receiving module and an analysis module; wherein,

[0117] The receiving module is configured to receive the control voice input by the user;

[0118] The analysis module is configured to analyze the control voice through a pre-trained voice recognition model to determine the voice control instruction.

[0119] Optionally, the device further includes a mixed reality module and a coordinate system construction module; wherein,

[0120] The mixed reality module is configured to capture a real-time reality image and generate a corresponding mixed reality image according to the real-time reality image;

[0121] The coordinate system construction module is configured to divide a two-dimensional coordinate system in the mixed reality image through a pre-set virtual two-dimensional coordinate system generation method to determine the user interaction image.

[0122] The voice positioning device based on a mixed reality device provided by an embodiment of the present invention can execute the voice positioning method based on a mixed reality device provided by any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution of the method.

[0123] Figure 5 FIG. shows a schematic structural diagram of a mixed reality device 10 that can be used to implement an embodiment of the present invention. The components shown herein, their connections and relationships, and their patterns are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0124] As Figure 5As shown, the mixed reality device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as read-only memory (ROM) 12, random access memory (RAM) 13, etc. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or the computer programs loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the mixed reality device 10 can also be stored. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0125] Multiple components in the mixed reality device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, optical disc, etc.; and a communication unit 19, such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the mixed reality device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor 11 can be various general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the voice localization method based on the mixed reality device.

[0127] In some embodiments, the voice localization method based on the mixed reality device can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the mixed reality device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the voice localization method based on the mixed reality device described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute the voice localization method based on the mixed reality device by any other appropriate means (e.g., by means of firmware).

[0128] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0129] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0130] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a mixed reality device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the mixed reality device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0133] A computing system can include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0134] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0135] This embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the voice positioning method based on a mixed reality device as provided in any embodiment of the present invention. The method includes:

[0136] In response to a voice control instruction for a user interaction image based on user input, determine a target coordinate corresponding to the voice control instruction;

[0137] Determine a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinate;

[0138] Mark the target image object in the user interaction image by a preset marking method, and display the positioning coordinate information.

[0139] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0140] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device.

[0141] The program code included on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0142] Computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or, alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0143] Those of ordinary skill in the art should understand that the various modules or steps of the present invention described above may be implemented with a general-purpose computing device. They may be concentrated on a single computing device or distributed across a network composed of multiple computing devices. Optionally, they may be implemented with program code executable by a computing device, so that they may be stored in a storage device and executed by the computing device, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them may be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0144] It should be understood that various forms of the flow shown above may be used, reordering, adding, or deleting steps. For example, the steps recited in the present invention may be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0145] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions may be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A voice localization method based on a mixed reality device, characterized in that including: responding to a voice control instruction for a user interaction image in response to user input, and determining a target coordinate corresponding to the voice control instruction; determining a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinate; marking the target image object in the user interaction image by a preset marking method, and displaying the positioning coordinate information.

2. The method according to claim 1, wherein The determining the target image object in the user interaction image and the positioning coordinate information corresponding to the target image object according to the target coordinate includes: scanning an image area corresponding to the target coordinate in the user interaction image to determine a target image object in the image area; performing coordinate positioning according to the target image object and the target coordinate to determine the positioning coordinate information.

3. The method according to claim 2, wherein The performing coordinate positioning according to the target image object and the target coordinate to determine the positioning coordinate information includes: performing image recognition on the target image object by a pre-trained large image recognition model to recognize target object information corresponding to the target image object; performing coordinate positioning according to the target object information and the target coordinate to determine the positioning coordinate information.

4. The method according to claim 3, characterized in that, The performing coordinate positioning according to the target object information and the target coordinate to determine the positioning coordinate information includes: combining the target object information and the target coordinate into object retrieval information; sending the object retrieval information to a mixed reality cloud for positioning search to determine the positioning coordinate information corresponding to the target image object.

5. The method according to claim 4, characterized in that, After combining the target object information and the target coordinate into object retrieval information, it further includes: sending the object retrieval information to a mixed reality cloud for information search to determine object display information corresponding to the target object information, and displaying the object display information in the user interaction image.

6. The method according to claim 1, characterized in that, Before responding to a voice control instruction for a user interaction image in response to user input and determining a target coordinate corresponding to the voice control instruction, it further includes: receiving a control voice input by the user; parsing the control voice by a pre-trained voice recognition model to determine the voice control instruction.

7. The method according to claim 1, characterized in that, Before responding to a voice control instruction for a user interaction image in response to user input and determining a target coordinate corresponding to the voice control instruction, it further includes: capturing a real-time reality image, and generating a corresponding mixed reality image according to the real-time reality image; performing two-dimensional coordinate system division in the mixed reality image by a pre-set virtual two-dimensional coordinate system generation method to determine a user interaction image.

8. A voice positioning device based on a mixed reality device, characterized in that, including: a voice module, configured to respond to a voice control instruction for a user interaction image in response to user input, and determine a target coordinate corresponding to the voice control instruction; a positioning module, configured to determine a target image object in the user interaction image and positioning coordinate information corresponding to the target image object according to the target coordinate; an object marking module, configured to mark the target image object in the user interaction image by a preset marking method, and display the positioning coordinate information.

9. A mixed reality device, characterized in that, The mixed reality device includes: at least one processor; and A memory communicatively connected to the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the voice localization method based on a mixed reality device according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for implementing the voice localization method based on a mixed reality device according to any one of claims 1-7 when the computer instructions are executed by a processor.

Images