Augmented reality image control method and device, storage medium and electronic equipment

Abstract

The invention relates to the field of augmented reality, in particular to an augmented reality image control method and device, a storage medium and electronic equipment. The augmented reality image control method comprises the steps of obtaining an augmented reality image, wherein the augmented reality image comprises target digital content obtained through an augmented reality technology; receiving a real-time voice instruction of a user; converting the real-time voice instruction into a control instruction based on a preset acoustic model; and controlling the target digital content in the augmented reality image according to the control instruction. By means of the method, the user can directly control the digital content on the mobile device only through the voice instruction, the interaction steps are simplified, the portability of user experience is improved, the interaction mode is innovated, and light weight of augmented reality interaction is achieved.

Description

Technical Field

[0001] The present disclosure relates to the field of augmented reality, and in particular to an augmented reality image control method, device, storage medium and electronic device. Background Art

[0002] With the advancement of augmented reality technology, there is an increasing demand for efficient and convenient human-computer interaction in many application scenarios. Although wearable devices, especially head-mounted devices, perform well in display effects and interactive experience, their convenience is still insufficient compared to mobile devices (such as smartphones, tablets, etc.). Mobile augmented reality technology came into being, which cleverly superimposes computer-generated virtual information or digital content onto the real scene observed by users through mobile devices.

[0003] Currently, mobile augmented reality mainly focuses on presenting digital content directly on the device screen and relies on touch and other contact methods for interaction. However, this single interaction mode limits its flexibility and diversity in complex scenarios and is difficult to meet diverse interaction needs. Summary of the invention

[0004] The main purpose of the present disclosure is to provide an augmented reality image control method, device, storage medium and electronic device, aiming to solve the technical problem of poor flexibility of the interaction mode of augmented reality technology in the prior art.

[0005] To achieve the above objectives, the present disclosure proposes an augmented reality image control method, comprising:

[0006] Acquire an augmented reality image, wherein the augmented reality image contains target digital content obtained by augmented reality technology;

[0007] Receive real-time voice commands from users;

[0008] Based on a preset acoustic model, convert the real-time voice command into a control command;

[0009] According to the control instruction, the target digital content in the augmented reality image is controlled.

[0010] Optionally, after controlling the target digital content in the augmented reality image according to the control instruction, the method further includes:

[0011] Acquiring a control result of the target digital content;

[0012] According to the control result, the augmented reality image is updated.

[0013] Optionally, acquiring the augmented reality image includes:

[0014] Capturing an image of the target area to obtain an image of the target area;

[0015] Acquire preset marker information, where the marker information includes an image template, feature points, and descriptors of at least one marker;

[0016] Performing feature matching on the target area image and the marker information to obtain a feature matching result;

[0017] In the case where the feature matching result satisfies a preset feature matching condition, determining, according to the feature matching result, a target marker corresponding to the target area image and initial spatial information of the target marker in the target area image;

[0018] The augmented reality image is determined according to initial spatial information of the target marker in the target area image.

[0019] Optionally, determining the augmented reality image according to initial spatial information of the target marker in the target area image includes:

[0020] Get initial virtual content;

[0021] Calculating the camera pose corresponding to the image acquisition device of the target area image according to the initial spatial information;

[0022] Determining target space information of the initial virtual content in the target area image according to the camera pose and the initial space information;

[0023] According to the target space information, the initial virtual content is projected into the target area image through coordinate transformation to obtain an augmented reality image containing the target digital content.

[0024] Optionally, performing feature matching on the target area image and the marker information to obtain a feature matching result includes:

[0025] Performing feature point detection on the target area image to obtain a first feature point set, wherein each feature point in the first feature point set corresponds to a descriptor; and

[0026] Performing feature point detection on the marker information to obtain a second feature point set, wherein each feature point in the second feature point set corresponds to a descriptor;

[0027] Perform feature matching on the descriptor of each feature point in the first feature point set and the descriptor of each feature point in the second feature point set to obtain a feature point matching pair set;

[0028] The feature matching result is determined according to the feature point matching pair set.

[0029] Optionally, converting the real-time voice command into a control command based on a preset acoustic model includes:

[0030] Based on the Mel-frequency cepstral coefficients, voice features are extracted from the voice command to obtain voice features contained in the voice command;

[0031] Inputting the speech feature into the acoustic model to obtain acoustic feature recognition information output by the acoustic model, wherein the acoustic feature recognition information includes phoneme information contained in the speech instruction and a score of each phoneme information;

[0032] Inputting the acoustic feature recognition information into a trained language model to obtain a text sequence corresponding to the voice command;

[0033] The text sequence is keyword matched with a preset instruction set to determine the control instruction.

[0034] Optionally, performing keyword matching on the text sequence and a preset instruction set to determine the control instruction includes:

[0035] Based on the regular expression, determine the word array corresponding to the text sequence;

[0036] Determine the command corresponding to the word-text array according to a preset word-text mapping table;

[0037] The control instruction is generated according to the command corresponding to the word array.

[0038] In addition, to achieve the above-mentioned purpose, the present disclosure further provides an augmented reality image control device, the augmented reality image control device comprising:

[0039] An acquisition module, used to acquire an augmented reality image, wherein the augmented reality image contains target digital content obtained by augmented reality technology;

[0040] A receiving module, used for receiving real-time voice commands from users;

[0041] A conversion module, used for converting the real-time voice command into a control command based on a preset acoustic model;

[0042] A control module is used to control the target digital content in the augmented reality image according to the control instruction.

[0043] In addition, to achieve the above objectives, the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and a processor executes the computer program to implement the above method.

[0044] In addition, to achieve the above-mentioned purpose, the present disclosure also provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned method.

[0045] In addition, to achieve the above objectives, the present disclosure also provides a computer program product, which implements the above method when executed by a processor.

[0046] Through the above technical solution, an augmented reality image is first obtained, and the augmented reality image contains the target digital content obtained by augmented reality technology. Then the user's real-time voice command is received, and based on the preset acoustic model, the real-time voice command is converted into a control command, which is used to perform specific processing or operation on the augmented reality image, and then the target digital content in the augmented reality image is controlled according to the control command. In this way, by combining real-time voice technology with augmented reality technology, the user can directly control the digital content on the mobile device to perform corresponding operations through voice commands, which greatly simplifies the interaction process and removes unnecessary physical operation steps. This voice-based interaction method not only improves the convenience of user experience, but also enhances the portability of mobile devices, and is no longer limited to specific physical controls or external devices. Compared with the traditional augmented reality technology that relies on clicking buttons to trigger digital content to perform actions, and the complex way that head-mounted devices need to use handles for contact interaction, mobile-end augmented reality technology based on real-time voice recognition has innovated the interaction mode and achieved lightweight augmented reality interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0048] Figure 1 A schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present disclosure;

[0049] Figure 2 A flowchart of an augmented reality image control method according to an embodiment of the present disclosure;

[0050] Figure 3 A schematic diagram of instruction matching involved in the embodiment of the present disclosure;

[0051] Figure 4 This is a structural block diagram of an augmented reality image control device involved in an embodiment of the present disclosure.

[0052] The realization of the objectives, functional features and advantages of the present disclosure will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0054] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present disclosure.

[0055] Typically, the device includes: at least one processor 301, a memory 302, and an augmented reality image control program stored in the memory 302 and executable on the processor 301, wherein the augmented reality image control program is configured to implement the steps of the augmented reality image control method as described above.

[0056] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. The processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process operations related to the augmented reality image control method, so that the augmented reality image control method model can be trained and learned autonomously to improve efficiency and accuracy.

[0057] The memory 302 may include one or more storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the augmented reality image control method provided in the method embodiment of the present disclosure.

[0058] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0059] The communication interface 303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0060] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 304 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0061] The display screen 305 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 305 is a touch display screen, the display screen 305 also has the ability to collect touch signals on the surface or above the surface of the display screen 305. The touch signal can be input to the processor 301 as a control signal for processing. At this time, the display screen 305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 305 can be one, the front panel of the electronic device; in other embodiments, the display screen 305 can be at least two, respectively arranged on different surfaces of the electronic device or in a folding design; in some embodiments, the display screen 305 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device. Even, the display screen 305 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 305 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0062] The power supply 306 is used to power various components in the electronic device. The power supply 306 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 306 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology. Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation of the device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0063] In addition, the embodiment of the present disclosure further proposes a storage medium, on which an augmented reality image control program is stored, and when the augmented reality image control program is executed by the processor, the steps of the augmented reality image control method described above are implemented. Therefore, it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the storage medium embodiment involved in the present disclosure, please refer to the description of the method embodiment of the present disclosure. As an example, the program instructions can be deployed to be executed on one device, or on multiple devices located at one location, or on multiple devices distributed at multiple locations and interconnected by a communication network.

[0064] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the above-mentioned program can be stored in a storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0065] Reference Figure 2 , Figure 2 The following is a flow chart of an augmented reality image control method according to an embodiment of the present disclosure, comprising the following steps:

[0066] Step S11: Acquire an augmented reality image, where the augmented reality image contains target digital content obtained through augmented reality technology.

[0067] Step S12: receiving the user's real-time voice instruction.

[0068] Step S13: based on a preset acoustic model, convert the real-time voice command into a control command.

[0069] Step S14: Control the target digital content in the augmented reality image according to the control instruction.

[0070] Through the above technical solution, an augmented reality image is first obtained, and the augmented reality image contains the target digital content obtained by augmented reality technology. Then the user's real-time voice command is received, and based on the preset acoustic model, the real-time voice command is converted into a control command, which is used to perform specific processing or operation on the augmented reality image, and then the target digital content in the augmented reality image is controlled according to the control command. In this way, by combining real-time voice technology with augmented reality technology, the user can directly control the digital content on the mobile device to perform corresponding operations through voice commands, which greatly simplifies the interaction process and removes unnecessary physical operation steps. This voice-based interaction method not only improves the convenience of user experience, but also enhances the portability of mobile devices, and is no longer limited to specific physical controls or external devices. Compared with the traditional augmented reality technology that relies on clicking buttons to trigger digital content to perform actions, and the complex way that head-mounted devices need to use handles for contact interaction, mobile-end augmented reality technology based on real-time voice recognition has innovated the interaction mode and achieved lightweight augmented reality interaction.

[0071] For example, voice can be collected through a mobile microphone device, for example, by calling the recording API provided by the mobile platform. Then configure the recording parameters, such as sampling rate, encoding format, etc., start recording, and receive the audio data captured by the microphone in real time. The collected audio data can also be digitally denoised, for example, by applying digital signal processing techniques such as FFT and filters to reduce background noise and echo. A special voice enhancement library can also be used to perform audio processing on the collected audio data. The disclosed embodiment does not limit the method of receiving and obtaining the user's voice instructions. In addition, the control of the target digital content in the augmented reality image depends on the content contained in the augmented reality image and the user's voice instructions.

[0072] Specifically, if the target digital content is a two-dimensional image, the control of the target digital content may include position and size adjustment, rotation and tilt, transparency adjustment, color and filter, etc., so that it can adapt to different viewing angles or user needs. If the target digital content is a three-dimensional model, the control of the target digital content may include position and posture control, animation and interaction, material and light and shadow adjustment, and splitting and combination, etc. Among them, position and posture control may include moving the position of the model in three-dimensional space, rotating the model around the X, Y, and Z axes, showing its views at different angles, and adjusting the size of the model, maintaining its proportion or performing non-uniform scaling. Animation and interaction adjustment may include adding animation effects to the model, such as rotation, jumping, flying, etc., to increase visual appeal. Animation and interaction may include enabling user interaction with the model, such as clicking, dragging, triggering events, etc. Material and light and shadow adjustment may include modifying the material of the model, such as metal, plastic, glass, etc., to affect the appearance and texture of the model. Adjusting light and shadow effects, such as adding light sources, shadows, reflections, etc., enhances the three-dimensional sense and realism of the model. Splitting and combination may include splitting multiple parts of the model and recombining these parts. The embodiments of the present disclosure do not specifically limit this.

[0073] In a possible manner, after controlling the target digital content in the augmented reality image according to the control instruction, the method further includes:

[0074] Acquiring control results of target digital content;

[0075] According to the control results, the augmented reality image is updated.

[0076] For example, in the present disclosure, after executing a control instruction on the target digital content, a feedback loop can be used to evaluate the control result and update the augmented reality image accordingly to ensure that the user's interaction with the augmented reality environment is immediate and responsive.

[0077] Specifically, after executing the control instruction, the control result of the target digital content can be obtained to capture the state change of the target digital content. The state change may include the update of attributes such as position, direction, size, color, transparency, etc. Based on the control result, the data used to render the augmented reality image is updated, which may specifically include updating the 3D model data, texture, material, lighting information, etc. of the target digital content. Then, the updated rendering data is used to regenerate and display the augmented reality image.

[0078] Among the possible ways to obtain the augmented reality image, include:

[0079] Capturing an image of the target area to obtain an image of the target area;

[0080] Acquire preset marker information, where the marker information includes an image template, feature points, and descriptors of at least one marker;

[0081] Perform feature matching on the target area image and the marker information to obtain a feature matching result;

[0082] When the feature matching result satisfies the preset feature matching condition, determining the target marker corresponding to the target area image and the initial spatial information of the target marker in the target area image according to the feature matching result;

[0083] An augmented reality image is determined according to initial spatial information of the target marker in the target area image.

[0084] It should be understood that in the present disclosure, one or more marker information may be predefined and stored, including image templates, feature points, and descriptors of the feature points of the markers. Specifically, image processing software or an algorithm library may be used to extract and store the feature points and descriptors of the markers.

[0085] For example, first use a camera or image sensor to shoot the designated target area to obtain a digital image of the area. Use a feature matching algorithm to match the features in the target area image with the preset marker features to determine the correspondence between the two. Specifically, feature points and descriptors can be extracted from the target area image first, and then these features can be compared with the preset marker features to obtain feature matching results. According to conditions such as the similarity of feature matching or the number of matching points, it is determined whether the feature matching result is valid. If valid, the markers contained in the target area image and their positions and directions in space, that is, the initial spatial information, are determined.

[0086] Finally, based on the initial spatial information of the marker, the corresponding augmented reality content is superimposed or rendered on the target area image to obtain an augmented reality image.

[0087] In a possible manner, determining an augmented reality image according to initial spatial information of a target marker in a target area image includes:

[0088] Get initial virtual content;

[0089] According to the initial spatial information, the camera pose corresponding to the image acquisition device of the target area image is calculated;

[0090] Determine the target space information of the initial virtual content in the target area image according to the camera pose and the initial space information;

[0091] According to the target space information, the initial virtual content is projected into the target area image through coordinate transformation to obtain an augmented reality image containing the target digital content.

[0092] For example, the initial virtual content to be displayed in the augmented reality scene is prepared in advance, which can be a 3D model, a 2D image, text, animation, etc. Specifically, a 3D model can be created using 3D modeling software, or a 2D image can be designed using image processing software. These resources are usually stored in the form of files and are ready to be loaded into the augmented reality application when needed.

[0093] Then, the initial spatial information of the target marker is used to infer the position and posture of the image acquisition device when taking the image of the target area, that is, the camera pose, through inverse perspective transformation or other geometric calculation methods.

[0094]

[0095] Specifically, the camera pose can be calculated by scanning the camera marker and the camera internal parameter matrix. Then, according to the target space information, the initial virtual content is projected into the target area image through coordinate transformation to obtain an augmented reality image containing the target digital content. The specific formula is as follows:

[0096] Among them, u i 、v i is the pixel coordinate of the digital content in the camera image plane, f x 、f y are the focal lengths of the camera in the horizontal and vertical directions, c x 、c y represents the center coordinate of the camera image, [R t] represents the rotation matrix and translation vector, X u , Y u , Z u Represent the three-dimensional coordinates of the markers respectively.

[0097] In a possible manner, feature matching is performed on the target area image and the marker information to obtain a feature matching result, including:

[0098] Performing feature point detection on the target area image to obtain a first feature point set, wherein each feature point in the first feature point set corresponds to a descriptor; and

[0099] Performing feature point detection on the marker information to obtain a second feature point set, wherein each feature point in the second feature point set corresponds to a descriptor;

[0100] Perform feature matching on the descriptor of each feature point in the first feature point set and the descriptor of each feature point in the second feature point set to obtain a feature point matching pair set;

[0101] The feature matching result is determined based on the feature point matching pair set.

[0102] For example, FAST feature point detection may be combined with BRIEF feature descriptors for feature matching.

[0103] The FAST feature point detection algorithm is as follows:

[0104]

[0105] Among them, S i is a ring of i consecutive pixels, i∈[1..16], I(x,y) is the brightness or grayscale value at the coordinate (x,y) in the image. S D The brighter part of the image center, S L is the darker part of the center point of the image, and d is the threshold. If the value of brightness or darkness of 12 consecutive pixels compared with the center point is greater than the threshold d, the center point is considered to be a feature point. p is used to measure the difference in brightness between the pixel and the center point, so that it can quickly detect feature points in the image. p(x,y) is the position of the detected feature point.

[0106] For each feature point in the first feature point set and each feature point in the second feature point set, a fixed pixel pair is randomly selected in the surrounding area. For each pair of pixels, their grayscale values ​​are compared. If the grayscale value of the first pixel is greater than that of the second pixel, it is recorded as 1, otherwise it is recorded as 0. After comparing multiple pixel pairs, binary data similar to "101101" can be generated, that is, a BRIEF feature descriptor is generated.

[0107] Then, the Hamming distance algorithm can be used to compare the similarity between all feature descriptors in the first feature point set and the second feature point set, that is, to determine the number of different 1 bits in the two feature descriptors. The Hamming distance algorithm is as follows:

[0108]

[0109] Among them, a i 、b iRepresents two binary strings of length i, n is the number of binary digits, 1(a i ≠b i ) is the indicator function. If a i ≠b i , then the indicator function 1(a i ≠b i ) is 1, otherwise it is 0. H(A,B) means a i 、b i The sum of the number of characters that differ between two binary strings at all corresponding positions. After comparing all feature descriptors, if the result does not exceed the threshold, the match is reliable.

[0110] Among possible ways, based on a preset acoustic model, real-time voice commands are converted into control commands, including:

[0111] Based on the Mel frequency cepstral coefficient, voice features of the voice command are extracted to obtain the voice features contained in the voice command;

[0112] Inputting the speech feature into the acoustic model to obtain acoustic feature recognition information output by the acoustic model, wherein the acoustic feature recognition information includes phoneme information contained in the speech instruction and a score of each phoneme information;

[0113] Input the acoustic feature recognition information into the trained language model to obtain the text sequence corresponding to the voice command;

[0114] The text sequence is keyword matched with a preset instruction set to determine the control instruction.

[0115] It should be understood that the acoustic model can map the input speech features (such as MFCCs) to a sequence of phonemes (or subword units) and give a likelihood score for each phoneme, which reflects the confidence of the model in recognizing each phoneme. This can be achieved using models such as deep neural networks (DNNs), recurrent neural networks (RNNs) or their variants (such as LSTM, GRU). These models are trained on a large amount of labeled data to learn the mapping relationship from speech features to phoneme sequences.

[0116] The language model is used to evaluate the language possibility of a given text sequence, that is, whether it conforms to the grammar and idiomatic usage of the language. You can use a statistical language model (such as the n-gram model) or a neural network language model (such as LSTM, Transformer, etc.). The language model is trained with a large amount of text data and can learn the statistical laws and contextual relationships of the language. The phoneme sequence output by the acoustic model is used as input, and the language model uses a decoding algorithm to find the most likely text sequence as output, that is, the text sequence corresponding to the voice command is obtained.

[0117] For example, the text sequence output by the language model is matched with a preset instruction set, and the control instruction is determined according to the matching result. If the text sequence completely matches a preset instruction or meets a certain similarity threshold, the instruction is output as the final control instruction.

[0118] In a possible manner, keyword matching of the text sequence with a preset instruction set to determine the control instruction includes:

[0119] Based on the regular expression, determine the word array corresponding to the text sequence;

[0120] Determine the command corresponding to the word-text array according to the preset word-text mapping table;

[0121] Generate control instructions according to the commands corresponding to the word array.

[0122] For example, keyword matching can be performed using regular expression matching technology. For example, the text "Please rotate 180 degrees" can be split into a verb "rotate" and a quantifier "180", and the regular expression pseudo code is:

[0123] const text = 'Please rotate 180 degrees';

[0124] const regexmotion = 'rotation';

[0125] const regexnum = '180';

[0126] const matchesmotion = text.match(regexmotion); / / matches 'rotation'

[0127] const matchesnum = text.match(regexnum); / / matches '180'

[0128] If both matchesmotion and matchesnum return data, it means that the complete match is successful.

[0129] Then, you can Figure 3 As shown, according to the preset word-text mapping table, the command corresponding to the word-text array is determined. The relationship between the word and the instruction can be expressed by a function y=f(x), where x is the input word content and y is the command corresponding to the word. Figure 3 The command for "rotation" is "rotate", so the command "cmd object rotate 180" can be generated, which means to rotate the digital content on the screen 180 degrees.

[0130] Reference Figure 4 , Figure 4 This is a structural block diagram of an augmented reality image control device involved in an embodiment of the present disclosure. Based on the same inventive concept as the above-mentioned embodiment, the device includes:

[0131] An acquisition module 10 is used to acquire an augmented reality image, wherein the augmented reality image includes target digital content obtained by augmented reality technology;

[0132] The receiving module 20 is used to receive the real-time voice command of the user;

[0133] A conversion module 30, configured to convert the real-time voice command into a control command based on a preset acoustic model;

[0134] The control module 40 is used to control the target digital content in the augmented reality image according to the control instruction.

[0135] Optionally, the augmented reality image control device further includes an updating module, configured to:

[0136] After controlling the target digital content in the augmented reality image according to the control instruction, obtaining a control result of the target digital content;

[0137] According to the control result, the augmented reality image is updated.

[0138] Optionally, the acquisition module 10 is used to:

[0139] Capturing an image of the target area to obtain an image of the target area;

[0140] Acquire preset marker information, where the marker information includes an image template, feature points, and descriptors of at least one marker;

[0141] Performing feature matching on the target area image and the marker information to obtain a feature matching result;

[0142] In the case where the feature matching result satisfies a preset feature matching condition, determining, according to the feature matching result, a target marker corresponding to the target area image and initial spatial information of the target marker in the target area image;

[0143] The augmented reality image is determined according to initial spatial information of the target marker in the target area image.

[0144] Optionally, the acquisition module 10 is used to:

[0145] Get initial virtual content;

[0146] Calculating the camera pose corresponding to the image acquisition device of the target area image according to the initial spatial information;

[0147] Determining target space information of the initial virtual content in the target area image according to the camera pose and the initial space information;

[0148] According to the target space information, the initial virtual content is projected into the target area image through coordinate transformation to obtain an augmented reality image containing the target digital content.

[0149] Optionally, the acquisition module 10 is used to:

[0150] Performing feature point detection on the target area image to obtain a first feature point set, wherein each feature point in the first feature point set corresponds to a descriptor; and

[0151] Performing feature point detection on the marker information to obtain a second feature point set, wherein each feature point in the second feature point set corresponds to a descriptor;

[0152] Perform feature matching on the descriptor of each feature point in the first feature point set and the descriptor of each feature point in the second feature point set to obtain a feature point matching pair set;

[0153] The feature matching result is determined according to the feature point matching pair set.

[0154] Optionally, the conversion module 30 is used to:

[0155] Based on the Mel-frequency cepstral coefficients, voice features are extracted from the voice command to obtain voice features contained in the voice command;

[0156] Inputting the speech feature into the acoustic model to obtain acoustic feature recognition information output by the acoustic model, wherein the acoustic feature recognition information includes phoneme information contained in the speech instruction and a score of each phoneme information;

[0157] Inputting the acoustic feature recognition information into a trained language model to obtain a text sequence corresponding to the voice command;

[0158] The text sequence is keyword matched with a preset instruction set to determine the control instruction.

[0159] Optionally, the conversion module 30 is used to:

[0160] Based on the regular expression, determine the word array corresponding to the text sequence;

[0161] Determine the command corresponding to the word-text array according to a preset word-text mapping table;

[0162] The control instruction is generated according to the command corresponding to the word array.

[0163] It should be noted that, since the steps executed by the device of this embodiment are the same as the steps of the aforementioned method embodiment, its specific implementation method and the technical effects that can be achieved can refer to the aforementioned embodiment and will not be repeated here.

[0164] In addition, in one embodiment, an embodiment of the present disclosure further provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory, and the computer program implements the steps of the method in the aforementioned embodiment when executed by the processor.

[0165] In addition, in one embodiment, an embodiment of the present disclosure further provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method in the aforementioned embodiment are implemented.

[0166] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0167] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0168] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0169] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0170] It should be noted that, in this article, the terms "include", "may include" or any other variations thereof are intended to cover non-exclusive inclusions, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0171] The serial numbers of the above-mentioned embodiments of the present disclosure are only for description and do not represent the advantages or disadvantages of the embodiments.

[0172] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0173] The above descriptions are only optional embodiments of the present disclosure, and are not intended to limit the patent scope of the present disclosure. All equivalent structural transformations made using the contents of the present disclosure and the drawings under the inventive concept of the present disclosure, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present disclosure.

Claims

1. An augmented reality image control method, characterized in that: include: Acquire an augmented reality image, wherein the augmented reality image contains target digital content obtained by augmented reality technology; Receive real-time voice commands from users; Based on a preset acoustic model, convert the real-time voice command into a control command; According to the control instruction, the target digital content in the augmented reality image is controlled.

2. The method according to claim 1, characterized in that After controlling the target digital content in the augmented reality image according to the control instruction, the method further includes: Acquiring a control result of the target digital content; According to the control result, the augmented reality image is updated.

3. The method according to claim 1, characterized in that: The acquiring of the augmented reality image comprises: Capturing an image of the target area to obtain an image of the target area; Acquire preset marker information, where the marker information includes an image template, feature points, and descriptors of at least one marker; Performing feature matching on the target area image and the marker information to obtain a feature matching result; In the case where the feature matching result satisfies a preset feature matching condition, determining, according to the feature matching result, a target marker corresponding to the target area image and initial spatial information of the target marker in the target area image; The augmented reality image is determined according to initial spatial information of the target marker in the target area image.

4. The method according to claim 3, characterized in that The step of determining the augmented reality image according to the initial spatial information of the target marker in the target area image comprises: Get initial virtual content; Calculating the camera pose corresponding to the image acquisition device of the target area image according to the initial spatial information; Determining target space information of the initial virtual content in the target area image according to the camera pose and the initial space information; According to the target space information, the initial virtual content is projected into the target area image through coordinate transformation to obtain an augmented reality image containing the target digital content.

5. The method according to claim 3, characterized in that: The performing feature matching on the target area image and the marker information to obtain a feature matching result includes: Performing feature point detection on the target area image to obtain a first feature point set, wherein each feature point in the first feature point set corresponds to a descriptor; and Performing feature point detection on the marker information to obtain a second feature point set, wherein each feature point in the second feature point set corresponds to a descriptor; Perform feature matching on the descriptor of each feature point in the first feature point set and the descriptor of each feature point in the second feature point set to obtain a feature point matching pair set; The feature matching result is determined according to the feature point matching pair set.

6. The method according to claim 1, characterized in that The converting the real-time voice command into a control command based on a preset acoustic model includes: Based on the Mel-frequency cepstral coefficients, voice features are extracted from the voice command to obtain voice features contained in the voice command; Inputting the speech feature into the acoustic model to obtain acoustic feature recognition information output by the acoustic model, wherein the acoustic feature recognition information includes phoneme information contained in the speech instruction and a score of each phoneme information; Inputting the acoustic feature recognition information into a trained language model to obtain a text sequence corresponding to the voice command; The text sequence is keyword matched with a preset instruction set to determine the control instruction.

7. The method according to claim 6, characterized in that The keyword matching of the text sequence with a preset instruction set to determine the control instruction includes: Based on the regular expression, determine the word array corresponding to the text sequence; Determine the command corresponding to the word-text array according to a preset word-text mapping table; The control instruction is generated according to the command corresponding to the word array.

8. An augmented reality image control device, characterized in that: include: An acquisition module, used to acquire an augmented reality image, wherein the augmented reality image contains target digital content obtained by augmented reality technology; A receiving module, used for receiving real-time voice commands from users; A conversion module, used for converting the real-time voice command into a control command based on a preset acoustic model; A control module is used to control the target digital content in the augmented reality image according to the control instruction.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 7.

Images