Augmented reality display processing method, apparatus, device, and storage medium

Abstract

The application provides a kind of augmented reality display processing method, device, electronic equipment and computer readable storage medium;Method includes: display reality scene, reality scene includes at least one real object;At least one candidate virtual object material associated with the feature of the feature of the real object in the augmented reality material package is matched, and the target feature of matching success is obtained;At least one virtual object material associated with the target feature and the material configuration file associated with the target feature are obtained from the augmented reality material package;At least one virtual object material is played based on the playing time of at least one virtual object material in the material configuration file, and is fused to at least one virtual object material of real object.By the application, the play of virtual object material can be flexibly controlled in augmented reality scene.

Description

Technical Field

[0001] This application relates to computer application technology, and more particularly to an augmented reality display processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the development of computer technology, electronic devices can realize richer and more vivid virtual scenes. A virtual scene refers to a digital scene outlined by a computer through digital communication technology. Users can obtain a fully virtualized experience (such as virtual reality) or a partially virtualized experience (such as augmented reality) in terms of vision, hearing, etc., and can also interact with various virtual objects in the virtual scene.

[0003] However, the related technologies have limited support for augmented reality display methods. For example, when users interact with augmented reality, they can only play virtual object materials through the programming logic of animation effects. This limitation on the playback of virtual object materials further affects the playback effect of virtual object materials. Summary of the Invention

[0004] This application provides an augmented reality display processing method, apparatus, electronic device, and computer-readable storage medium, which can flexibly control the playback of virtual object materials in augmented reality scenes.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an augmented reality display processing method, including:

[0007] Display a real-world scene, wherein the real-world scene includes at least one real object;

[0008] The features associated with at least one candidate virtual object in the augmented reality material package are matched with the features of the real object to obtain the target features that are successfully matched.

[0009] Obtain at least one virtual object material associated with the target feature and a material configuration file associated with the target feature from the augmented reality material package;

[0010] Based on the playback timing of the at least one virtual object material in the material configuration file, the at least one virtual object material that is integrated into the real object is played.

[0011] This application provides an augmented reality display processing device, comprising:

[0012] A display module is used to display a real-world scene, the real-world scene including at least one real object;

[0013] The matching module is used to match the features associated with at least one candidate virtual object material in the augmented reality material package with the features of the real object to obtain the target features that are successfully matched.

[0014] The acquisition module is used to acquire at least one virtual object material associated with the target feature and a material configuration file associated with the target feature from the augmented reality material package;

[0015] The playback module is used to play the at least one virtual object material that is integrated into the real object based on the playback timing of the at least one virtual object material in the material configuration file.

[0016] In the above technical solution, the matching module is further configured to perform the following processing for each candidate virtual object material in the at least one candidate virtual object material: obtain a first feature map corresponding to the feature associated with the candidate virtual object material;

[0017] Obtain the second feature map corresponding to the feature points on the surface of the real object;

[0018] For each candidate virtual object, the first feature map and the second feature map are subjected to feature point matching processing, and the first feature map that is successfully matched is taken as the target feature that is successfully matched.

[0019] In the above technical solution, the display module is further configured to display a matching failure prompt message when the feature associated with the at least one candidate virtual object material fails to match the feature of the real object;

[0020] The matching failure message indicates that the real-world scene needs to be reshot.

[0021] In the above technical solution, the display module is further used to trigger the client's matching failure handling logic, so as to display the prompt information through the matching failure handling logic; or,

[0022] The trigger state machine transitions to a feature matching failure state to trigger the display of virtual object material associated with the feature matching failure state, wherein the virtual object material associated with the feature matching failure state includes the matching failure prompt information.

[0023] In the above technical solution, the virtual object material includes at least one of the following: virtual object model, virtual object model animation, and multimedia file;

[0024] The playback module is also configured to perform at least one of the following operations:

[0025] During the time period that satisfies the playback timing of the virtual object model, the virtual object model is superimposed and displayed on the surface of the real object;

[0026] During the time period that satisfies the playback timing of the virtual object model animation, the virtual object model animation is played on the surface of the real object;

[0027] The multimedia file is played during a time period that satisfies the playback timing of the multimedia file.

[0028] In the above technical solution, the device further includes:

[0029] The processing module is used to bind the playback timing of at least one virtual object material in the material configuration file, and the triggering events to be executed within the time period that meets the playback timing, to the state respectively to obtain a state machine;

[0030] When a target feature that has been successfully matched is obtained through the matching process, the state machine is set to the initial state;

[0031] The time periods that satisfy different playback timings and different triggering events are determined by the switching of states in the state machine.

[0032] The triggering event is used to trigger the playback of at least one virtual object material that is integrated into the real object, based on the effect parameters in the material configuration file.

[0033] In the above technical solution, the playback module is further configured to determine the plane corresponding to the target feature in the real scene when a successfully matched target feature is obtained through the matching process, and establish a mapping relationship between the world coordinate system and the screen coordinate system based on the plane as a reference plane.

[0034] Based on the mapping relationship, the virtual object model in the world coordinate system is mapped to the surface of the real object in the screen coordinate system.

[0035] In the above technical solution, the playback timing of the virtual object model includes at least one of the following: displaying the start time of the virtual object model, displaying the duration of the virtual object model, and interactive operations on the virtual object model;

[0036] The playback timing of the virtual object model animation includes at least one of the following: the start time of playing the virtual object model animation, the duration of playing the virtual object model animation, and interactive operations on the virtual object model animation;

[0037] The playback timing of the multimedia file includes at least one of the following: the start time of playing the multimedia file, the duration of playing the multimedia file, the number of times the multimedia file is looped, and interactive operations on the multimedia file.

[0038] In the above technical solution, the display module is also used to display multiple candidate virtual object materials in the augmented reality material package;

[0039] In response to the selection operation for the plurality of candidate virtual object materials, the features associated with the selected candidate virtual object material are used as the features associated with the at least one candidate virtual object material for performing the matching process.

[0040] In the above technical solution, the display module is further used to perform predictive processing on the multiple candidate virtual object materials through a neural network model to obtain the user's preference for the multiple candidate virtual object materials;

[0041] Based on the user's preference for the multiple candidate virtual object materials, the multiple candidate virtual object materials are sorted in descending order;

[0042] The multiple candidate virtual object materials are displayed based on the descending sorting result.

[0043] In the above technical solution, the display module is also used to acquire the interactive parameters of the plurality of candidate virtual object materials;

[0044] Based on the interaction parameters of the multiple candidate virtual object materials, the multiple candidate virtual object materials are sorted in descending order;

[0045] The multiple candidate virtual object materials are displayed based on the descending sorting result.

[0046] In the above technical solution, the display module is also used to obtain the usage frequency of the plurality of candidate virtual object materials;

[0047] The candidate virtual object materials are displayed in descending order of usage frequency.

[0048] In the above technical solution, the display module is also used to display guidance information for the selected candidate virtual object material;

[0049] The guidance information is used to indicate the conditions for shooting the real-world scene when playing the selected candidate virtual object material.

[0050] In the above technical solution, the device further includes:

[0051] A configuration module is used to respond to a triggering operation of the configuration entry of the material configuration file associated with the selected candidate virtual object material;

[0052] The configuration interface displays the configuration file associated with the selected candidate virtual object material;

[0053] In response to a configuration operation on the configuration interface, the material configuration file associated with the selected candidate virtual object material is updated based on the configuration parameters input in the configuration operation.

[0054] This application provides an electronic device for display processing in augmented reality, the electronic device comprising:

[0055] Memory, used to store executable instructions;

[0056] The processor, when executing executable instructions stored in the memory, implements the augmented reality display processing method provided in the embodiments of this application.

[0057] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the augmented reality display processing method provided in this application.

[0058] The embodiments of this application have the following beneficial effects:

[0059] Based on the playback timing of virtual object materials in the material configuration file, virtual object materials that are integrated into real objects are played, thereby flexibly controlling the playback of virtual object materials through the material configuration file and achieving diverse playback effects of virtual object materials. Attached Figure Description

[0060] Figure 1A-Figure 1B This is a schematic diagram illustrating the application mode of the augmented reality display processing method provided in the embodiments of this application;

[0061] Figure 2 This is a schematic diagram of the structure of an electronic device for display processing of augmented reality provided in an embodiment of this application;

[0062] Figures 3A-3B This is a schematic flowchart of the augmented reality display processing method provided in the embodiments of this application;

[0063] Figure 4 This is a schematic diagram of a real object provided in the embodiments of this application;

[0064] Figures 5A-5B This is a schematic diagram of an augmented reality scene provided in an embodiment of this application;

[0065] Figure 6This is a schematic diagram illustrating the relationships provided in the embodiments of this application;

[0066] Figure 7 This is a schematic diagram of the matching failure prompt information provided in the embodiments of this application;

[0067] Figure 8 This is a schematic diagram of the candidate virtual object material provided in the embodiments of this application;

[0068] Figure 9 This is a schematic diagram of the guidance information provided in the embodiments of this application;

[0069] Figure 10 This is the configuration interface provided in the embodiments of this application;

[0070] Figures 11A-11C This is a schematic diagram illustrating the effect of playing virtual object materials provided in the embodiments of this application;

[0071] Figure 12 This is a schematic diagram of the design side provided in an embodiment of this application;

[0072] Figure 13 This is a schematic flowchart of the augmented reality display processing method provided in the embodiments of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0074] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0076] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0077] 1) Virtual scenes: These are scenes that differ from the real world, which are output by devices. Visual perception of virtual scenes can be formed with the naked eye or with the assistance of devices. For example, two-dimensional images are output through a display screen, and three-dimensional images are output through stereoscopic display technologies such as stereoscopic projection, virtual reality, and augmented reality. In addition, various possible hardware can be used to form various perceptions that simulate the real world, such as auditory perception, tactile perception, olfactory perception, and motion perception.

[0078] 2) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0079] 3) Client: Applications running on the terminal that provide various services, such as game clients, short video clients, etc.

[0080] 4) Virtual object material: Any element in a virtual scene, including at least one of the following: virtual object model, virtual object model animation, multimedia file (audio, video), such as characters (roles) and props in a game, where users can control characters or props to fight against other users; people and objects in virtual reality (VR); virtual effects in augmented reality (AR).

[0081] 5) Virtual Object Model: This refers to the interactive images of various people and objects within a virtual scene, or movable objects within the virtual scene. These movable objects can be virtual characters, virtual animals, cartoon characters, etc., such as people, animals, plants, oil drums, walls, and stones displayed in the virtual scene. A virtual object can be a virtual avatar representing the user within the virtual scene. A virtual scene can include multiple virtual objects, each with its own shape and volume, occupying a portion of the space within the virtual scene.

[0082] 6) Marker Augmented Reality (AR): An augmented reality technology that first creates markers in advance, such as QR codes or feature maps of target objects. By recognizing the markers in the image, a mapping relationship between the world coordinate system and the screen coordinate system is established to achieve augmented reality functions.

[0083] In related technologies, the gameplay of AR effects (i.e. virtual object materials) in shooting is relatively simple. It involves clicking on the screen or placing a three-dimensional model on a marker, while playing background music or animation. The timing of the playback cannot be customized, and the effects that can be achieved are limited. It is not possible to create complex and detailed effects with a sense of layering, which restricts the creative expression of designers.

[0084] The applicant found that the AR effects in the relevant technologies are limited. Most only support placing virtual models (i.e., virtual object models) into the AR scene at the start of shooting based on the results of AR processing, and playing a background song at the same time. However, it cannot support placing multiple virtual models or playing multiple songs, nor can it control the timing of the appearance of virtual models and music. This shortcoming limits the expressiveness of the effects, and can only produce some relatively simple special effects. Furthermore, the development of AR effects in the relevant technologies is complex. Each AR effect requires additional logic to be written, and there is no universal and configurable solution. It is difficult for designers to adjust the effects independently, and developers need to be deeply involved, resulting in low output efficiency.

[0085] To address the aforementioned problems, embodiments of this application provide an augmented reality display processing method, apparatus, electronic device, and computer-readable storage medium, capable of flexibly controlling the playback of virtual object materials through material configuration files. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as various types of user terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or as a server. The following will describe exemplary applications when the device is implemented as a terminal.

[0086] To facilitate a clearer understanding of the augmented reality display processing method provided in this application, an exemplary implementation scenario of the augmented reality display processing method provided in this application is first described. The augmented reality virtual scene can be output entirely based on the terminal, or output based on the collaboration between the terminal and the server.

[0087] In one implementation scenario, see Figure 1A , Figure 1A This is a schematic diagram of the application mode of the augmented reality display processing method provided in the embodiments of this application. It is applicable to some application modes that can complete the relevant data calculation of the virtual scene 100 by relying entirely on the computing power of the terminal 400. The output of the augmented reality virtual scene is completed by the terminal 400 such as smartphones, tablets and augmented reality devices.

[0088] When visual perception of the augmented reality virtual scene 100 is formed, the terminal 400 calculates the data required for display through graphics computing hardware, and completes the loading, parsing and rendering of the display data. The graphics output hardware outputs video frames that can form visual perception of augmented reality. For example, two-dimensional video frames are presented on the display screen of a smartphone, or video frames that achieve a three-dimensional display effect are projected onto the lenses of augmented reality glasses. In addition, in order to enrich the perception effect, the device can also use different hardware to form one or more of auditory perception, tactile perception, motion perception and taste perception.

[0089] As an example, terminal 400 runs client 410 with augmented reality functionality (e.g., standalone short video application, live streaming application, instant messaging application, video editing application, etc.). During the operation of client 410, the output includes an augmented reality virtual scene 100 (including real objects and virtual object models). The virtual scene includes virtual object models 110, which can be props controlled by the user (or player). That is, virtual object models 110 are controlled by the real user and will operate in the virtual scene in response to the real user's gestures. The virtual scene includes real objects 120, which can be real-world items, such as banknotes.

[0090] In another implementation scenario, see Figure 1B , Figure 1B This is a schematic diagram of the application mode of the augmented reality display processing method provided in the embodiments of this application. It is applied to the terminal 400 and the server 200, and is suitable for the application mode that relies on the computing power of the server 200 to complete the calculation of the augmented reality virtual scene and output the augmented reality virtual scene on the terminal 400.

[0091] Taking the visual perception of an augmented reality virtual scene 100 as an example, the server 200 calculates the display data related to the augmented reality virtual scene and sends it to the terminal 400. The terminal 400 relies on graphics computing hardware to load, parse, and render the calculated display data, and relies on graphics output hardware to output the virtual scene to form visual perception. For example, two-dimensional video frames can be displayed on the screen of a smartphone, or video frames that achieve a three-dimensional display effect can be projected onto the lenses of augmented reality glasses. As for the perception of the form of the augmented reality virtual scene, it can be understood that it can be achieved with the help of the corresponding hardware output of the terminal, such as using a microphone to form auditory perception, using a vibrator to form tactile perception, and so on.

[0092] As an example, terminal 400 runs client 410 with augmented reality functionality (such as a web-based short video application, live streaming application, instant messaging application, video editing application, etc.). By connecting to the server of the short video application (i.e., server 200), it interacts with other users in games. Terminal 400 outputs augmented reality virtual scene 100 (including real objects and virtual object models) of client 410. The virtual scene includes virtual object model 110, which can be a prop controlled by the user (or player). That is, virtual object model 110 is controlled by the real user and will operate in the virtual scene in response to the real user's gestures. The virtual scene includes real object 120, which can be real-world items, such as banknotes.

[0093] In some embodiments, the terminal 400 can implement the augmented reality display processing method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a short video APP (i.e., the client 410 mentioned above); it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a game mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0094] The embodiments of this application can be implemented with the help of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and network within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0095] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0096] As an example, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, or smartwatch, but is not limited to these. Terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0097] The structure of the electronic device for display processing of augmented reality provided in the embodiments of this application is described below. See also... Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device for display processing of augmented reality provided in an embodiment of this application. The description takes the electronic device as a terminal as an example. Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0098] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0099] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0100] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may include, for example, one or more storage devices physically located away from the processor 410.

[0101] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0102] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0103] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0104] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0105] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0106] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0107] In some embodiments, the augmented reality display processing apparatus provided in this application can be implemented in software. Figure 2 An augmented reality display processing device 455 stored in memory 450 is shown. It can be software in the form of programs and plug-ins, including the following software modules: display module 4551, matching module 4552, acquisition module 4553, playback module 4554, processing module 4555, and configuration module 4556. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0108] As mentioned above, the augmented reality display processing method provided in this application embodiment can be implemented by various types of electronic devices. See also Figure 3A , Figure 3A This is a flowchart illustrating the augmented reality display processing method provided in the embodiments of this application, combined with... Figure 3A The steps shown are explained.

[0109] It should be noted that, Figure 3A The method shown can be executed by various forms of computer programs running on terminal 400, and is not limited to the client 410 described above, such as the operating system 451, software modules, scripts and applets mentioned above. Therefore, the client example in the following text should not be regarded as a limitation on the embodiments of this application.

[0110] In step 101, a real-world scene is displayed, which includes at least one real object.

[0111] For example, an image of a real-world scene is captured by a terminal with augmented reality capabilities. This image includes images of real objects within the scene, and the captured scene is displayed on the terminal's screen. The captured real-world scene includes at least one real object. See also... Figure 4 The real world is photographed through a terminal with augmented reality capabilities, and the banknote 402 in the real scene 401 is displayed on the terminal's display interface.

[0112] Specifically, based on the viewing position and field of view of the user, the field of view area of ​​the viewing object is determined, and a portion of the virtual scene located within the field of view area is presented. That is, the displayed virtual scene can be a portion of the virtual scene relative to the panoramic virtual scene.

[0113] For example, taking the example of a user wearing an augmented reality device, see Figure 5A , Figure 5A This is a schematic diagram of the augmented reality interface provided in this application embodiment. The viewing user (i.e., the real user) can perceive a virtual scene 502, including the real scene 501, through the lens in the augmented reality device. The augmented reality device is equipped with a posture detection sensor (such as a nine-axis sensor) to detect the posture changes of the augmented reality device in real time. If the user is wearing the augmented reality device, when the user's head posture changes, the real-time head posture will be transmitted to the processor to calculate the user's gaze point in the virtual scene. Based on the gaze point, the image in the three-dimensional model of the virtual scene that is within the user's gaze range (i.e., the field of view area) is calculated and displayed on the display screen, giving people an immersive experience as if they were in a real environment.

[0114] Taking the example of a user controlling an augmented reality device to display a real-world scene 501, where the viewing user is a real person 504 within the complete real-world scene 503, see [link / reference]. Figure 5B , Figure 5B This is a schematic diagram of the augmented reality interface provided in the embodiment of this application. A real person 504 controls the augmented reality device and adjusts the viewing position and field of view in the complete real scene 503 to present a part of the real scene 505 in the complete real scene 503.

[0115] In step 102, the features associated with at least one candidate virtual object material in the augmented reality material package are matched with the features of the real object to obtain the target features that are successfully matched.

[0116] For example, after displaying a real-world scene, multiple candidate virtual object assets in the augmented reality (AR) asset package can be automatically matched with real objects. This involves matching the features of the candidate virtual object assets with the features of the real objects to obtain the successfully matched target features. The candidate virtual object associated with the target feature becomes the target virtual object asset. The target feature itself may not be in the AR asset package but in a feature library independent of it, thus saving storage resources; alternatively, it may be in the AR asset package, for example, in the asset configuration file.

[0117] In some embodiments, the features associated with at least one candidate virtual object material in the augmented reality material package are matched with the features of a real object to obtain a successfully matched target feature. This includes performing the following processing for each candidate virtual object material in the at least one candidate virtual object material: obtaining a first feature map corresponding to the features associated with the candidate virtual object material; obtaining a second feature map corresponding to the feature points on the surface of the real object; performing feature point matching processing on the first feature map and the second feature map of each candidate virtual object, and using the successfully matched first feature map as the successfully matched target feature.

[0118] For example, a feature map includes feature vectors used to describe feature points. The first feature map and the second feature map are compared using similarity analysis to obtain the similarity between the features associated with the candidate virtual object material and the real object. When the similarity is greater than a similarity threshold, the feature associated with the candidate virtual object material corresponding to that similarity is determined as the target feature.

[0119] like Figure 6 As shown, an image 601 (feature) of a 5-yuan note is associated with an AR effect 602 (a candidate virtual object material) that captures a 5-yuan note. When the image 601 associated with the AR effect 602 matches a real object (a real 5-yuan banknote), the image 601 is taken as the target feature of the successful match.

[0120] In some embodiments, after matching the features associated with at least one candidate virtual object material in the augmented reality material package with the features of the real object, a matching failure prompt message can be displayed when the features associated with at least one candidate virtual object material fail to match the features of the real object; wherein, the matching failure prompt message is used to indicate that the real scene needs to be reshot.

[0121] For example, if the features associated with all candidate virtual objects in the augmented reality asset package do not match the features of the real object, it means that the current real object is not applicable to all candidate virtual objects in the augmented reality asset package, and a matching failure message can be displayed on the display interface of the electronic device.

[0122] like Figure 7 As shown, a 5-yuan banknote 402 in the real-world scene 401 is displayed on the terminal's display interface. When the features associated with all the candidate virtual object materials in the augmented reality material package do not match the 5-yuan banknote 402, a matching failure prompt message 403 is displayed on the terminal's display interface to prompt the user to retake the photo.

[0123] Following the example above, there are two ways to display the matching failure message: 1) Trigger the client's matching failure handling logic to display the message through the matching failure handling logic; 2) Trigger the state machine in the client to transition to the feature matching failure state to trigger the display of the virtual object material associated with the feature matching failure state. The virtual object material associated with the feature matching failure state includes the matching failure message. Since the matching failure message included in the virtual object material can be customized, the matching failure message can be flexibly modified.

[0124] Following the example above, to facilitate resetting the matching failure message at any time, a setting entry for the matching failure message is displayed on the terminal's display interface. In response to a trigger operation on the setting entry, the setting interface for the matching failure message is displayed on the terminal's display interface. In response to a setting operation on the setting interface, the setting parameters in the setting interface are updated to update the matching failure message.

[0125] In some embodiments, the terminal may allow users to manually select candidate virtual object assets and match the features associated with the selected candidate virtual object assets with real objects. For example, multiple candidate virtual object assets from an augmented reality asset package are displayed on the display interface; in response to the selection operation of multiple candidate virtual object assets, the features associated with the selected candidate virtual object assets are used as features associated with at least one candidate virtual object asset for matching processing.

[0126] like Figure 8As shown, multiple candidate virtual object materials are presented on the terminal's display interface, such as shooting AR effects worth 20 yuan, shooting AR effects of exclusive masks, etc. When the user selects to shoot AR effects worth 50 yuan 801, the terminal will use the features associated with shooting AR effects worth 50 yuan 801 as the target features, and then the AR effects worth 50 yuan 801 can be presented on the display interface.

[0127] Following the example above, displaying multiple candidate virtual object assets in an augmented reality asset package can be achieved in the following way: using a neural network model to predict the multiple candidate virtual object assets to obtain the user's preference for the multiple candidate virtual object assets; sorting the multiple candidate virtual object assets in descending order based on the user's preference for the multiple candidate virtual object assets; and displaying the multiple candidate virtual object assets according to the descending order result.

[0128] The neural network model can be a deep neural network, a convolutional neural network, or similar. For example, the neural network model predicts multiple candidate virtual object materials based on the user's profile information to obtain the user's preference for each candidate virtual object material. Based on the user's preference for multiple candidate virtual object materials, the multiple candidate virtual object materials are sorted in descending order, and the top 10 candidate virtual object materials in the descending order are displayed on the terminal's display interface for the user to select.

[0129] Following the example above, displaying multiple candidate virtual object assets in an augmented reality asset package can be achieved in the following way: obtaining the interaction parameters of multiple candidate virtual object assets; sorting the multiple candidate virtual object assets in descending order based on the interaction parameters; and displaying the multiple candidate virtual object assets according to the descending order result.

[0130] For example, interaction parameters represent user interaction information with candidate virtual object materials, such as the number of likes, shares, and comments. Based on the interaction parameters of multiple candidate virtual object materials, the materials are sorted in descending order, and the top 10 materials in the sorted order are displayed on the terminal's interface for the user to select.

[0131] Following the example above, displaying multiple candidate virtual object assets in an augmented reality asset package can be achieved by: obtaining the usage frequency of multiple candidate virtual object assets; and displaying multiple candidate virtual object assets in descending order of usage frequency.

[0132] For example, by obtaining the frequency of user usage of multiple candidate virtual object assets, a higher usage frequency indicates that the user likes that candidate virtual object asset more. Similarly, by obtaining the usage frequency of sample users of multiple candidate virtual object assets, a higher usage frequency indicates that the candidate virtual object asset is more popular, and users are more likely to like it.

[0133] In some embodiments, after the selected candidate virtual object material is determined, guidance information for the selected candidate virtual object material may also be displayed; wherein the guidance information is used to indicate the conditions for shooting the real scene when playing the selected candidate virtual object material.

[0134] like Figure 9 As shown, multiple candidate virtual object materials are presented on the terminal's display interface, such as AR effects for shooting 20 yuan, AR effects for shooting a special mask, etc. After the user selects to shoot 50 yuan AR effect 801, guidance information 802 for shooting 50 yuan AR effect 801 can be displayed on the display interface, such as "Please shoot the back of the banknote", to remind the user to take a normal shot and avoid matching the target features, so that the shooting 50 yuan AR effect 801 cannot be displayed.

[0135] In step 103, at least one virtual object material associated with the target feature and a material configuration file associated with the target feature are obtained from the augmented reality material package.

[0136] The material configuration file includes effect parameters, which include display effect parameters of the virtual object model (such as display position, animation style, animation duration, etc.) and playback effect parameters of multimedia files (audio, video) (such as duration, number of loop playbacks, etc.).

[0137] In some embodiments, after determining candidate virtual object materials, the terminal may also support manual adjustment of the material configuration file, for example, in response to a user's triggering operation on the configuration entry of the material configuration file associated with the selected candidate virtual object material; displaying the configuration interface of the material configuration file associated with the selected candidate virtual object material; and in response to the user's configuration operation on the configuration interface, updating the material configuration file associated with the selected candidate virtual object material based on the configuration parameters input in the configuration operation.

[0138] like Figure 10As shown, after the user selects to shoot the AR effect 801 worth 50 yuan, they can click the configuration entry 803 of the material configuration file associated with the AR effect 801 worth 50 yuan. Then, the configuration interface 804 of the material configuration file associated with the AR effect 801 worth 50 yuan will be displayed on the display screen. The user can enter configuration parameters in the configuration interface to update the material configuration file associated with the AR effect 801 worth 50 yuan. In other words, the user can design AR effects according to their own preferences.

[0139] The media configuration file can be set before playing at least one virtual object media that is integrated into a real object, during playing virtual object media, or after playing at least one virtual object media that is integrated into a real object.

[0140] In step 104, based on the playback timing of at least one virtual object material in the material configuration file, at least one virtual object material that is integrated into the real object is played.

[0141] For example, after obtaining virtual object materials associated with the target feature and material configuration files associated with the target feature from the augmented reality material package, when the playback timing of the virtual object material in the material configuration file is triggered, at least one virtual object material that is integrated into the real object is played to achieve diverse augmented reality display effects.

[0142] See Figure 3B , Figure 3B This is an optional flowchart illustrating an augmented reality display processing method provided in an embodiment of this application. Figure 3B Show Figure 3A Step 104 can also be implemented by at least one of the following steps: in step 1041, during the time period that satisfies the playback timing of the virtual object model, the virtual object model is overlaid and displayed on the surface of the real object; in step 1042, during the time period that satisfies the playback timing of the virtual object model animation, the virtual object model animation is played on the surface of the real object; in step 1043, during the time period that satisfies the playback timing of the multimedia file, the multimedia file is played.

[0143] The virtual object materials include at least one of the following: virtual object models, virtual object model animations, and multimedia files. The playback timing of virtual object models includes at least one of the following: the start time of displaying the virtual object model, the duration of displaying the virtual object model, and interactive operations performed on the virtual object model. The playback timing of virtual object model animations includes at least one of the following: the start time of playing the virtual object model animation, the duration of playing the virtual object model animation, and interactive operations performed on the virtual object model animation. The playback timing of multimedia files includes at least one of the following: the start time of playing the multimedia file, the duration of playing the multimedia file, the number of times the multimedia file is looped, and interactive operations performed on the multimedia file. Interactive operations can be gestures (e.g., clicking), voice, body sensing, facial expressions, eye movements (e.g., watching for 1 second), etc.

[0144] As an example, the playback timing of the virtual object model can be the start time of displaying the virtual object model, the duration of displaying the virtual object model, or an interactive operation on the virtual object model. For example, the playback timing of the virtual object model can be the start time 1 second after matching the target feature (the corresponding playback time period of the virtual object model can be from 1 second after matching the target feature to the total playback duration of the virtual object material or the time period after turning off the playback of the virtual object material), continuously displayed for 2 seconds (the corresponding playback time period of the virtual object model can be within 2 seconds after matching the target feature), or when a click gesture is detected (the corresponding playback time period of the virtual object model can be from the start gesture of detecting the click gesture to the end gesture of detecting the double-click gesture), etc.

[0145] As an example, the timing of the virtual object model's playback can be any combination of the start time of displaying the virtual object model, the duration of displaying the virtual object model, and interactive operations on the virtual object model. For example, the virtual object model's playback timing could be the start time 1 second after matching the target feature and the display lasting for 2 seconds (the corresponding playback timing time period could be 1 to 3 seconds after matching the target feature). The virtual object model's playback timing could also be the start time 1 second after matching the target feature and the detection of a click gesture (the corresponding playback timing time period could be from 1 second after matching the target feature to the detection of the click gesture), and so on.

[0146] like Figure 11AAs shown, the real world is photographed by a terminal with augmented reality function. After the banknote 402 (real object) in the real scene 401 is presented in the display interface of the terminal, after 1 second (satisfying the playback timing of displaying mountain peak 403, that is, the start time of displaying virtual object model), mountain peak 403 is superimposed on the surface of banknote 402 until the AR effect ends (the playback timing of mountain peak 403 is 1 second until the AR effect ends).

[0147] like Figure 11B As shown, after the mountain peak 403 is superimposed on the surface of the banknote 402, a click gesture is detected (satisfying the playback timing of the Phoenix 404 animation, i.e., an interactive operation on the virtual object model). The Phoenix 404 flying animation is played around the banknote 402 until a double-click gesture is detected (the playback timing of Phoenix 404 is from the detection of the click gesture to the detection of the double-click gesture).

[0148] like Figure 11C As shown, while an animation of a phoenix 404 flying around banknote 402 is played, background music 405 is played twice (the timing of playing background music 405 is the beginning of the animation of the phoenix 404 flying).

[0149] In some embodiments, before playing at least one virtual object material merged into a real object based on the playback timing of at least one virtual object material in the material configuration file, the playback timing of at least one virtual object material in the material configuration file and the trigger events to be executed within the time period that meet the playback timing are respectively bound to a state to obtain a state machine; when a target feature that is successfully matched is obtained through matching processing, the state machine is set to the initial state; the time period that meets different playback timings and different trigger events are determined by the switching of states in the state machine; wherein, the trigger events are used to trigger the playback of at least one virtual object material merged into a real object based on the effect parameters in the material configuration file.

[0150] The effect parameters include: display effect parameters of the virtual object model (such as display position, animation style, animation duration, etc.) and playback effect parameters of the multimedia file (such as duration, number of loop playbacks, etc.). Specifically, the playback timing of each virtual object material in the material configuration file, and the trigger events to be executed within the time period that meets the playback timing, are bound to their corresponding states, resulting in a state machine bound to different states.

[0151] For example, when a target feature is matched, the state machine is set to the initial state. Subsequently, the time periods that meet different playback timings and different trigger events can be determined by switching states in the state machine. Based on the determined trigger events, the effect parameters in the material configuration file are determined, thereby triggering the execution of playback of the virtual object material that is integrated into the real object.

[0152] Following the example above, displaying a virtual object model overlaid on the surface of a real object includes: when a target feature is successfully matched through matching processing, determining the plane corresponding to the target feature in the real scene, using the plane as a reference plane, and establishing a mapping relationship between the world coordinate system and the screen coordinate system based on the reference plane; and based on the mapping relationship, mapping the virtual object model in the world coordinate system onto the surface of the real object in the screen coordinate system.

[0153] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0154] This application proposes an augmented reality display processing method. Based on the processing results of Marker AR capabilities, the surface of an object is determined according to the processing results. Multiple 3D virtual models are placed on the surface of the object, and the timing of the virtual models, music, and animation playback can be controlled to form an immersive experience that is integrated with the captured image.

[0155] like Figure 12 As shown, designers create corresponding 3D virtual models and animations bound to them based on the theme and content of the AR effect, saving them in a common model format (e.g., GLB) without additional model file conversion. To make the AR effect look natural and vivid, designers add lighting and shadows to the 3D model. Designers can add corresponding fields and parameters to the material configuration file to control the color, direction, intensity of the lights, and whether shadows are enabled. After completing the initial design work, the design resources (including the 3D virtual model and background music) and the material configuration file (including AR effect parameters, such as the animation parameters of the 3D virtual model) are placed together in the corresponding material directory on the client side to preview the effect. This allows designers to independently adjust and modify the effect without developer involvement, improving the efficiency of effect production.

[0156] For example, such as Figure 9 As shown, the client integrates six AR effects using embodiments of this application, all of which involve shooting banknotes, corresponding to denominations of 100 yuan, 50 yuan, 20 yuan, 10 yuan, 5 yuan, and 1 yuan. Figure 11BAs shown, place the banknote face down on the table, click on the AR effect corresponding to the denomination, and take a picture. A scene model 403 will appear on the banknote, along with an animation effect that grows out of the paper. After a second, a phoenix 404 flies in from the corner of the screen and hovers around the scene model 403, while music plays.

[0157] like Figure 13 As shown below, the augmented reality display processing method proposed in this application includes three stages, as detailed in the following embodiments:

[0158] Phase 1: Place objects according to the gameplay and prepare a feature map of the object surface for each AR effect (i.e., the AR effect is associated with the feature map, and the feature map is also associated with the configuration file). Match the feature points of the images captured by the camera with the prepared feature map.

[0159] The feature map can be an image of the target object's surface from a frontal angle or a grayscale image. Multiple feature maps are allowed, meaning matching and detection can be performed on various surfaces, but only the first matched surface will be returned. The first stage extracts features from the captured image, obtaining feature points and feature vectors describing those feature points, and then performs a matching operation with the corresponding features in the feature map. After a successful match, the feature map is placed in a real-world location (i.e., placed on the object's surface) to define a plane in the real-world scene. A world coordinate system is established with the center of the feature map as the origin, the feature map surface as the XY-axis plane, and the normal vector perpendicular to this XY-axis plane as the Z-axis. Then, the feature map's pose is evaluated using a camera to obtain a mapping transformation from the world coordinate system to the screen coordinate system. Based on this mapping transformation, a virtual model drawn on the screen can achieve the effect of being placed on the object's surface.

[0160] When a match is successful, the AR effect associated with the matched feature map is determined, indicating that the AR effect matches the object surface and can present the associated AR effect.

[0161] Phase 2: If the match fails, the effect will not be triggered. At this time, you can prompt the user that the effect will only occur when a specific object is scanned. If the match is successful, a world coordinate system is established with the surface of the matched object as the XY plane and the normal vector perpendicular to the plane as the Z axis. The origin is the geometric center of the object surface. The 3D virtual model bound to the feature map will be placed in the corresponding position in the world coordinate system according to the coordinates defined in the material configuration file (associated with the successfully matched feature map).

[0162] The matching result differentiation process is divided into two cases: 1) matching failure and 2) matching success.

[0163] 1) Matching Failure: The matching result will be returned to the client layer along with a matching failure message. This message can be handled by pre-defined logic in the client or configured in the assets, triggering a corresponding state to display it. The message can consist of text or sticker animations.

[0164] 2) Match Successful: Each feature map has a unique identifier used to determine which type of object surface was detected from the captured image. In some scenarios, it's necessary to identify and distinguish between multiple object surfaces. For example, in a banknote photography scenario, an AR effect is designed on the back of the banknote, but users unfamiliar with the process might shoot at the front. In this case, a prompt message needs to be displayed to guide the user to shoot at the back of the banknote. After obtaining the identifier, the client-side upper-layer logic determines whether it is an object surface from the real-world scene. If it is not, a prompt message is displayed; if it is, the state machine is set to the start state, triggering the overall effect playback.

[0165] Phase Three: Upon successful matching, the program sets the state machine to the start state, indicating that the effect begins playback. Subsequent state transitions are triggered according to the playback timing specified in the material configuration file (for example, when the playback timing of different events is set at certain time intervals, the state machine triggers different events at those intervals, i.e., an event is triggered after a specified period of time). Whenever the state machine transitions to the next state, the triggering action (i.e., the triggering event) bound to that state is executed. Triggering actions can include placing 3D virtual models, playing animations bound to 3D virtual models, playing background music, etc. The execution times of different events are diverse and can be mutually exclusive, parallel, or interleaved.

[0166] To support complex animation effects through configuration, state machine state transitions are used to trigger effects. Trigger events can include placing a 3D virtual model at a specific position in the world coordinate system, playing animation effects bound to the 3D virtual model, or playing background music. Each trigger event is associated with a state in the state machine; when the state machine transitions to the corresponding state, the trigger event is executed. AR effects appear after the feature map matches the surface of the target object. Therefore, upper-level client logic needs to set the state machine to the start state after detecting the target object in the shooting frame. Subsequent state transitions are time-driven; for example, waiting 2 seconds to enter state 1, placing the 3D virtual model on the surface of the real-world object, then waiting another 3 seconds to enter state 2, and starting to play background music. Through time-based state transitions, this embodiment achieves timing control of various effects. Furthermore, the time of state transitions and the corresponding effects can be configured in JSON format in the material configuration file, improving the efficiency of special effects development.

[0167] In summary, the augmented reality display processing method proposed in this application has the following beneficial effects:

[0168] 1) Rich effects: Supports placing multiple 3D virtual models on the surfaces of objects recognized in the shooting scene. Each 3D virtual model can have its own unique animation effects. The timing of the 3D virtual model appearing in the frame and the timing of the animation playback can be freely adjusted to milliseconds. The entire effect is triggered based on whether a specific object surface is recognized. Multiple background music tracks can be configured for the effects, and the timing of the music playback within the effect can be controlled.

[0169] 2) Reduce the development workload when producing new effects. Designers can set various attributes in the material configuration file, such as the appearance time of the 3D virtual model, the start time and duration of the 3D virtual model animation, the start time and loop count of the background music, etc., and package them together with the corresponding design resources to generate a new shooting scene AR effect without additional development work, making it possible to mass-produce similar effects and speeding up the release of effects.

[0170] 3) It supports complex timing relationships between 3D virtual models, animations, and background music, enhancing the expressiveness of the effects. By combining state transitions and configuration files, it reduces the development workload when producing new special effects and improves creative efficiency.

[0171] The exemplary application and implementation of the terminal provided in the embodiments of this application have been used to describe the augmented reality display processing method provided in the embodiments of this application. The following will continue to describe the scheme of how the various modules in the augmented reality display processing device 455 provided in the embodiments of this application cooperate to realize the augmented reality display processing.

[0172] Display module 4551 is used to display a real-world scene, the real-world scene including at least one real object; matching module 4552 is used to match the features associated with at least one candidate virtual object material in the augmented reality material package with the features of the real object to obtain a successfully matched target feature; acquisition module 4553 is used to acquire at least one virtual object material associated with the target feature and a material configuration file associated with the target feature from the augmented reality material package; playback module 4554 is used to play the at least one virtual object material merged into the real object based on the playback timing of the at least one virtual object material in the material configuration file.

[0173] In some embodiments, the matching module 4552 is further configured to perform the following processing for each candidate virtual object material in the at least one candidate virtual object material: obtain a first feature map corresponding to the feature associated with the candidate virtual object material; obtain a second feature map corresponding to the feature points of the surface of the real object; perform feature point matching processing on the first feature map and the second feature map of each candidate virtual object, and take the first feature map that is successfully matched as the target feature that is successfully matched.

[0174] In some embodiments, the display module 4551 is further configured to display a matching failure prompt when the features associated with the at least one candidate virtual object material fail to match the features of the real object; wherein the matching failure prompt is used to indicate that the real scene needs to be reshot.

[0175] In some embodiments, the display module 4551 is further configured to trigger the client's matching failure handling logic to display the prompt information through the matching failure handling logic; or, to trigger a state machine to transition to a feature matching failure state to trigger the display of virtual object material associated with the feature matching failure state, wherein the virtual object material associated with the feature matching failure state includes the matching failure prompt information.

[0176] In some embodiments, the virtual object material includes at least one of the following: a virtual object model, a virtual object model animation, and a multimedia file; the playback module 4554 is further configured to perform at least one of the following operations: displaying the virtual object model overlaid on the surface of the real object during a time period that satisfies the playback timing of the virtual object model; playing the virtual object model animation on the surface of the real object during a time period that satisfies the playback timing of the virtual object model animation; and playing the multimedia file during a time period that satisfies the playback timing of the multimedia file.

[0177] In some embodiments, the apparatus further includes: a processing module 4555, configured to bind the playback timing of the at least one virtual object material in the material configuration file and the triggering event to be executed within the time period that satisfies the playback timing to a state, respectively, to obtain a state machine; when a target feature that is successfully matched is obtained through the matching process, the state machine is set to an initial state; the time period that satisfies different playback timings and different triggering events are determined by the switching of states in the state machine; wherein the triggering event is used to trigger the playback of the at least one virtual object material that is blended into the real object based on the effect parameters in the material configuration file.

[0178] In some embodiments, the playback module 4554 is further configured to, when a successfully matched target feature is obtained through the matching process, determine the plane corresponding to the target feature in the real scene, use the plane as a reference plane, establish a mapping relationship between the world coordinate system and the screen coordinate system based on the reference plane, and map the virtual object model in the world coordinate system to the surface of the real object in the screen coordinate system based on the mapping relationship.

[0179] In some embodiments, the playback timing of the virtual object model includes at least one of the following: displaying the start time of the virtual object model, displaying the duration of the virtual object model, and interactive operations on the virtual object model; the playback timing of the virtual object model animation includes at least one of the following: playing the start time of the virtual object model animation, playing the duration of the virtual object model animation, and interactive operations on the virtual object model animation; the playback timing of the multimedia file includes at least one of the following: playing the start time of the multimedia file, playing the duration of the multimedia file, looping the multimedia file a certain number of times, and interactive operations on the multimedia file.

[0180] In some embodiments, the display module 4551 is further configured to display a plurality of candidate virtual object materials in the augmented reality material package; in response to a selection operation for the plurality of candidate virtual object materials, the features associated with the selected candidate virtual object material are used as features associated with the at least one candidate virtual object material for performing the matching process.

[0181] In some embodiments, the display module 4551 is further configured to perform predictive processing on the plurality of candidate virtual object materials through a neural network model to obtain the user's preference for the plurality of candidate virtual object materials; sort the plurality of candidate virtual object materials in descending order based on the user's preference for the plurality of candidate virtual object materials; and display the plurality of candidate virtual object materials according to the result of the descending sort.

[0182] In some embodiments, the display module 4551 is further configured to acquire the interaction parameters of the plurality of candidate virtual object materials; sort the plurality of candidate virtual object materials in descending order based on the interaction parameters of the plurality of candidate virtual object materials; and display the plurality of candidate virtual object materials according to the result of the descending sort.

[0183] In some embodiments, the display module 4551 is further configured to obtain the usage frequency of the plurality of candidate virtual object materials; and display the plurality of candidate virtual object materials in descending order of the usage frequency.

[0184] In some embodiments, the display module 4551 is further configured to display guidance information for the selected candidate virtual object material; wherein the guidance information is used to indicate the conditions for shooting the real scene when playing the selected candidate virtual object material.

[0185] In some embodiments, the apparatus further includes: a configuration module 4556, configured to: respond to a triggering operation of a configuration entry for a configuration file associated with the selected candidate virtual object material; display a configuration interface for the configuration file associated with the selected candidate virtual object material; and, in response to a configuration operation on the configuration interface, update the configuration file associated with the selected candidate virtual object material based on configuration parameters input by the configuration operation.

[0186] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the augmented reality display processing method described above in this application.

[0187] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the augmented reality display processing method provided in this application. For example, ... Figures 3A-3B The augmented reality display processing method is shown.

[0188] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0189] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0190] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0191] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0192] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An augmented reality display processing method, characterized in that, The method includes: Display a real-world scene, wherein the real-world scene includes at least one real object; The features associated with at least one candidate virtual object in the augmented reality material package are matched with the features of the real object to obtain the target features that are successfully matched. Obtain at least one virtual object material associated with the target feature and a material configuration file associated with the target feature from the augmented reality material package, wherein the execution time relationship between the virtual object materials is mutually exclusive, parallel, or interleaved; The playback timing of at least one virtual object material in the material configuration file, and the triggering event to be executed within the time period that satisfies the playback timing, are respectively bound to the state to obtain a state machine. The playback timing of the virtual object material includes at least one of the start time, duration, or number of loop playbacks of the virtual object material. When the target feature that has been successfully matched is obtained through the matching process, the state machine is set to the initial state; The time periods that satisfy different playback timings and different trigger events are determined by the state transitions in the state machine. The state transitions of the state machine after the initial state are time-driven. When the state machine transitions to the next state, the trigger event bound to the next state is executed to achieve timing control of the start time, duration, or number of loop playbacks. Based on the playback timing of the at least one virtual object material in the material configuration file, the at least one virtual object material merged into the real object is played, wherein the triggering event is used to trigger the playback based on the effect parameters in the material configuration file.

2. The method according to claim 1, characterized in that, The step of matching the features associated with at least one candidate virtual object material in the augmented reality material package with the features of the real object to obtain the successfully matched target features includes: For each candidate virtual object material in the at least one candidate virtual object material, the following process is performed: obtain the first feature map corresponding to the feature associated with the candidate virtual object material; Obtain the second feature map corresponding to the feature points on the surface of the real object; For each candidate virtual object, the first feature map and the second feature map are subjected to feature point matching processing, and the first feature map that is successfully matched is taken as the target feature that is successfully matched.

3. The method according to claim 1, characterized in that, After matching the features associated with at least one candidate virtual object material in the augmented reality material package with the features of the real object, the method further includes: When the feature associated with the at least one candidate virtual object material fails to match the feature of the real object, a matching failure message is displayed; The matching failure message indicates that the real-world scene needs to be reshot.

4. The method according to claim 3, characterized in that, The message indicating a matching failure includes: Trigger the client's matching failure handling logic to display the prompt message through the matching failure handling logic; or, The trigger state machine transitions to a feature matching failure state to trigger the display of virtual object material associated with the feature matching failure state, wherein the virtual object material associated with the feature matching failure state includes the matching failure prompt information.

5. The method according to claim 1, characterized in that, The virtual object material includes at least one of the following: virtual object model, virtual object model animation, and multimedia file; Playing the at least one virtual object material integrated into the real object based on the playback timing of the at least one virtual object material in the material configuration file includes: Perform at least one of the following operations: During the time period that satisfies the playback timing of the virtual object model, the virtual object model is superimposed and displayed on the surface of the real object; During the time period that satisfies the playback timing of the virtual object model animation, the virtual object model animation is played on the surface of the real object; The multimedia file is played during a time period that satisfies the playback timing of the multimedia file.

6. The method according to claim 5, characterized in that, The process of overlaying and displaying the virtual object model on the surface of the real object includes: When a target feature is successfully matched through the matching process, the plane corresponding to the target feature in the real scene is determined. The plane is used as a reference plane, and a mapping relationship between the world coordinate system and the screen coordinate system is established based on the reference plane. Based on the mapping relationship, the virtual object model in the world coordinate system is mapped to the surface of the real object in the screen coordinate system.

7. The method according to claim 5, characterized in that, The playback timing of the virtual object model includes at least one of the following: displaying the start time of the virtual object model, displaying the duration of the virtual object model, and interactive operations on the virtual object model; The playback timing of the virtual object model animation includes at least one of the following: the start time of playing the virtual object model animation, the duration of playing the virtual object model animation, and interactive operations on the virtual object model animation; The playback timing of the multimedia file includes at least one of the following: the start time of playing the multimedia file, the duration of playing the multimedia file, the number of times the multimedia file is looped, and interactive operations on the multimedia file.

8. The method according to claim 1, characterized in that, Before matching the features associated with at least one candidate virtual object asset in the augmented reality asset package with the features of the real object, the method further includes: Displays multiple candidate virtual object assets from the augmented reality asset package; In response to the selection operation for the plurality of candidate virtual object materials, the features associated with the selected candidate virtual object material are used as the features associated with the at least one candidate virtual object material for performing the matching process.

9. The method according to claim 8, characterized in that, The display of multiple candidate virtual object materials in the augmented reality material package includes: The multiple candidate virtual object materials are predicted using a neural network model to obtain the user's preference for the multiple candidate virtual object materials; Based on the user's preference for the multiple candidate virtual object materials, the multiple candidate virtual object materials are sorted in descending order; The multiple candidate virtual object materials are displayed based on the descending sorting result.

10. The method according to claim 8, characterized in that, The display of multiple candidate virtual object materials in the augmented reality material package includes: Obtain the interaction parameters of the multiple candidate virtual object materials; Based on the interaction parameters of the multiple candidate virtual object materials, the multiple candidate virtual object materials are sorted in descending order; The multiple candidate virtual object materials are displayed based on the descending sorting result.

11. The method according to claim 8, characterized in that, The display of multiple candidate virtual object materials in the augmented reality material package includes: Obtain the usage frequency of the multiple candidate virtual object materials; The candidate virtual object materials are displayed in descending order of usage frequency.

12. The method according to claim 8, characterized in that, After the features associated with the candidate virtual object material to be selected are used as features associated with the at least one candidate virtual object material for performing the matching process, the method further includes: Display guidance information for the selected candidate virtual object material; The guidance information is used to indicate the conditions for shooting the real-world scene when playing the selected candidate virtual object material.

13. The method according to claim 8, characterized in that, After the features associated with the candidate virtual object material to be selected are used as features associated with the at least one candidate virtual object material for performing the matching process, the method further includes: In response to a triggering operation of the configuration entry of the material configuration file associated with the selected candidate virtual object material; The configuration interface displays the configuration file associated with the selected candidate virtual object material; In response to a configuration operation on the configuration interface, the material configuration file associated with the selected candidate virtual object material is updated based on the configuration parameters input in the configuration operation.

14. An augmented reality display processing device, characterized in that, The device includes: A display module is used to display a real-world scene, the real-world scene including at least one real object; The matching module is used to match the features associated with at least one candidate virtual object material in the augmented reality material package with the features of the real object to obtain the target features that are successfully matched. The acquisition module is used to acquire at least one virtual object material associated with the target feature and a material configuration file associated with the target feature from the augmented reality material package, wherein the execution time relationship between the virtual object materials is mutually exclusive, parallel, or interleaved. The processing module is used to bind the playback timing of at least one virtual object material in the material configuration file and the trigger event to be executed within the time period that satisfies the playback timing to a state, thereby obtaining a state machine. The playback timing of the virtual object material includes at least one of the start time, duration, or number of loop playbacks of the virtual object material. When the target feature that is successfully matched is obtained through the matching process, the state machine is set to the initial state. The time period that satisfies different playback timings and different trigger events are determined by the switching of states in the state machine. The state transition of the state machine after the initial state is time-driven. When the state machine transitions to the next state, the trigger event bound to the next state is executed to realize the timing control defined by the start time, the duration, or the number of loop playbacks. A playback module is used to play at least one virtual object material integrated into the real object based on the playback timing of at least one virtual object material in the material configuration file, wherein the triggering event is used to trigger the playback based on the effect parameters in the material configuration file.

15. The apparatus according to claim 14, characterized in that, The matching module is also used for: For each candidate virtual object material in the at least one candidate virtual object material, the following process is performed: obtain the first feature map corresponding to the feature associated with the candidate virtual object material; Obtain the second feature map corresponding to the feature points on the surface of the real object; For each candidate virtual object, the first feature map and the second feature map are subjected to feature point matching processing, and the first feature map that is successfully matched is taken as the target feature that is successfully matched.

16. The apparatus according to claim 14, characterized in that, The display module is also used for: After matching the features associated with at least one candidate virtual object material in the augmented reality material package with the features of the real object, a matching failure prompt message is displayed when the features associated with the at least one candidate virtual object material fail to match the features of the real object; wherein, the matching failure prompt message is used to indicate that the real scene needs to be reshot.

17. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the augmented reality display processing method according to any one of claims 1 to 13.

18. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the augmented reality display processing method according to any one of claims 1 to 13 when executed by a processor.

19. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the augmented reality display processing method according to any one of claims 1 to 13.

Images