Method and device for displaying special effect images

Abstract

The present application discloses a method and device for displaying special effects pictures, belonging to the field of artificial intelligence technology. The method includes: displaying a scene image captured by a camera device; determining a marker selected in the scene image, the marker has associated virtual reality special effects information, and when the shooting angle of the camera device changes, the association between the marker and the virtual reality special effects information remains unchanged; displaying a special effects picture, the special effects picture includes virtual reality special effects information associated with the marker displayed on the scene image. The present application selects markers in the scene image, improves the convenience of marker selection, utilizes the association between the markers in the scene image and the virtual reality special effects information, and the display angle of the virtual reality special effects information is adapted to the shooting angle of the camera device, which can present a more realistic special effects picture, and is applied to various scenes that require special effects pictures, such as being applied to a video tool that generates a special effects picture, to improve the user's interactive experience.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a method and device for displaying special effect images. Background Art

[0002] Virtual reality is a process of superimposing virtual models onto real-world scene images to obtain corresponding special effects images, that is, presenting the real environment and virtual models in the same picture.

[0003] At present, virtual reality technology mainly superimposes fixed virtual models and real scene images. The superposition effect is relatively simple, which affects the user experience. Summary of the invention

[0004] The embodiment of the present application provides a method and device for displaying special effect images, which can be used to solve the problems in related technologies. The technical solution is as follows:

[0005] On the one hand, an embodiment of the present application provides a method for displaying a special effects screen, the method comprising: displaying a scene image captured by a camera device; determining a marker selected in the scene image, the marker having associated virtual reality special effects information, and when the shooting angle of the camera device changes, the association between the marker and the virtual reality special effects information remains unchanged; displaying a special effects screen, the special effects screen including the virtual reality special effects information associated with the marker displayed on the scene image, and the display angle of the virtual reality special effects information being adapted to the shooting angle of the camera device.

[0006] In a possible implementation, the special effects picture also includes a processing control, and the processing control includes at least one of a sharing control, a publishing control or a modification control, the sharing control is used to share the special effects picture, the publishing control is used to publish the special effects picture, and the modification control is used to modify the special effects picture; after displaying the special effects picture, it also includes: when the processing control is triggered, performing a processing operation corresponding to the processing control on the special effects picture.

[0007] In a possible implementation, before displaying the special effects screen, it also includes: displaying multiple model resources uploaded by the user, determining a selected model resource from the multiple model resources as a virtual model associated with the marker; performing special effects processing on the virtual model to obtain virtual reality special effects information for rendering the virtual model, and associating the virtual reality special effects information with the marker.

[0008] In a possible implementation, the virtual reality special effect information includes first virtual reality special effect information or second virtual reality special effect information, and the special effect processing of the virtual model to obtain virtual reality special effect information for rendering the virtual model may include: acquiring model parameters of the virtual model, and determining dimension information of the virtual model based on the model parameters; when the dimension information indicates that the virtual model is a two-dimensional virtual model, decoding and format conversion processing are performed on the virtual model to obtain the first virtual reality special effect information; when the dimension information indicates that the virtual model is a three-dimensional virtual model, determining format information of the three-dimensional virtual model based on the model parameters, and processing the virtual model based on the format information to obtain the second virtual reality special effect information.

[0009] In a possible implementation, the decoding and format conversion of the virtual model to obtain the first virtual reality special effect information may include: decoding the virtual model using a decoder to obtain a frame image in a first image format, where the first image format is an image format output by the decoder; and format conversion of the frame image in the first image format to obtain the first virtual reality special effect information.

[0010] In a possible implementation, the processing the virtual model based on the format information to obtain the second virtual reality special effect information may include: determining the first virtual model or the second virtual model based on the format information and the standard format information of the three-dimensional virtual model, the format information of the first virtual model is the same as the standard format information, and the format information of the second virtual model is different from the standard format information; when the three-dimensional virtual model is the first virtual model, parsing the first virtual model to obtain the second virtual reality special effect information or parsing error information corresponding to the first virtual model; performing format conversion on the first virtual model corresponding to the parsing error information to obtain the second virtual reality special effect information; when the three-dimensional virtual model is the second virtual model, performing format conversion on the second virtual model to obtain the second virtual reality special effect information.

[0011] In a possible implementation, after determining the selected marker in the scene image, it also includes: performing feature matching between multiple scene images and the marker; obtaining rendering parameters based on the successfully matched scene image and the marker; rendering the marker and the virtual model based on the rendering parameters to obtain a rendered picture; and superimposing the rendered picture with the scene image to obtain the special effects picture.

[0012] In a possible implementation, the feature matching of the multiple scene images and the marker may include: obtaining a feature matching interval, determining multiple first scene images and multiple second scene images in the multiple scene images based on the feature matching interval, the first scene image being determined based on the feature matching interval, and the second scene image being located between the first scene images; based on the generation order of the scene images, feature matching the first first scene image with the marker to obtain first matching information, the first matching information including the matching degree between the first first scene image and the marker and the position information of a reference feature; based on the first matching information, tracking and detecting the position of the camera device corresponding to the marker and the second scene image to obtain second matching information, the second matching information including the moving position information of the reference feature; based on the second matching information, feature matching the first scene images other than the first first scene image with the marker to obtain third matching information, the third matching information including the matching degree between the first scene image and the marker.

[0013] In a possible implementation, the virtual reality special effects information includes model parameters of the virtual model, and obtaining rendering parameters based on the successfully matched scene image and the marker includes: obtaining parameters of the camera device based on the successfully matched scene image and the marker; determining a first coordinate system corresponding to the marker using the parameters of the camera device, and determining a second coordinate system corresponding to the virtual model using the model parameters of the virtual model; obtaining parameter information of the first coordinate system and the second coordinate system, and processing at least one of the parameters of the camera device or the model parameters based on the parameter information to obtain rendering parameters.

[0014] In a possible implementation, the processing of the parameters of the camera device or at least one of the model parameters based on the parameter information to obtain rendering parameters may include: when the directions of the coordinate axes of the first coordinate system and the second coordinate system are the same, determining a parameter matrix corresponding to the parameters of the camera device and the model parameters, and obtaining the rendering parameters based on the parameter matrix; when the directions of the coordinate axes of the first coordinate system and the second coordinate system are different, determining conversion parameters of the first coordinate system and the second coordinate system, adjusting the parameters of the camera device or at least one of the model parameters based on the conversion parameters, determining a parameter matrix corresponding to the adjusted parameters of the camera device or at least one of the model parameters, and obtaining the rendering parameters based on the parameter matrix.

[0015] In a possible implementation, superimposing the rendering screen with the scene image to obtain the special effects screen may include: superimposing the rendering screen with the scene image to obtain a preview screen; obtaining a control instruction, wherein the control instruction is used to confirm the preview screen; and using the preview screen as the special effects screen according to the control instruction.

[0016] In a possible implementation, obtaining the control instruction may include: detecting at least one of the first trigger information or the second trigger information, and generating the corresponding control instruction when at least one of the first trigger information or the second trigger information is detected, wherein the first trigger information is generated based on a control of a preview interface, the preview interface includes the preview screen, and the second trigger information is generated based on at least one of a user's hand posture or a hand motion trajectory.

[0017] On the other hand, an embodiment of the present application provides a display device for a special effects screen, the device comprising: a first display module, used to display a scene image captured by a camera device; a determination module, used to determine a marker selected in the scene image, the marker having associated virtual reality special effects information, and when the shooting angle of the camera device changes, the association between the marker and the virtual reality special effects information remains unchanged; a second display module, used to display a special effects screen, the special effects screen including the virtual reality special effects information associated with the marker displayed on the scene image, and the display angle of the virtual reality special effects information is adapted to the shooting angle of the camera device.

[0018] In a possible implementation, the special effects picture also includes a processing control, and the processing control includes at least one of a sharing control, a publishing control or a modification control, the sharing control is used to share the special effects picture, the publishing control is used to publish the special effects picture, and the modification control is used to modify the special effects picture; the second display module is also used to perform a processing operation corresponding to the processing control on the special effects picture when the processing control is triggered.

[0019] In a possible implementation, the determination module is used to display multiple model resources uploaded by a user, determine a selected model resource from the multiple model resources as a virtual model associated with the marker; perform special effects processing on the virtual model to obtain virtual reality special effects information for rendering the virtual model, and associate the virtual reality special effects information with the marker.

[0020] In a possible implementation, the virtual reality special effect information includes first virtual reality special effect information and second virtual reality special effect information, and the determination module is used to obtain model parameters of the virtual model, and determine dimension information of the virtual model based on the model parameters; when the dimension information indicates that the virtual model is a two-dimensional virtual model, the virtual model is decoded and format converted to obtain the first virtual reality special effect information; when the dimension information indicates that the virtual model is a three-dimensional virtual model, the format information of the three-dimensional virtual model is determined based on the model parameters, and the virtual model is processed based on the format information to obtain the second virtual reality special effect information.

[0021] In a possible implementation, the determination module is used to use a decoder to decode the virtual model to obtain a frame image in a first image format, where the first image format is an image format output by the decoder; and to perform format conversion on the frame image in the first image format to obtain the first virtual reality special effect information.

[0022] In a possible implementation, the determination module is used to determine the first virtual model or the second virtual model based on the format information and standard format information of the three-dimensional virtual model, the format information of the first virtual model is the same as the standard format information, and the format information of the second virtual model is different from the standard format information; when the three-dimensional virtual model is the first virtual model, the first virtual model is parsed to obtain the second virtual reality special effect information or parsing error information corresponding to the first virtual model; the first virtual model corresponding to the parsing error information is format converted to obtain the second virtual reality special effect information; when the three-dimensional virtual model is the second virtual model, the second virtual model is format converted to obtain the second virtual reality special effect information.

[0023] In a possible implementation, the determination module is further used to perform feature matching between multiple scene images and the marker; obtain rendering parameters based on the successfully matched scene images and the marker; render the marker and the virtual model based on the rendering parameters to obtain a rendered picture; and superimpose the rendered picture with the scene image to obtain the special effects picture.

[0024] In a possible implementation, the determination module is used to obtain a feature matching interval, and determine multiple first scene images and multiple second scene images in the multiple scene images based on the feature matching interval, wherein the first scene image is determined based on the feature matching interval, and the second scene image is located between the first scene images; based on the generation order of the scene images, feature matching is performed on the first first scene image and the marker to obtain first matching information, and the first matching information includes the matching degree between the first first scene image and the marker and the position information of the reference feature; based on the first matching information, the position of the camera device corresponding to the marker and the second scene image is tracked and detected to obtain second matching information, and the second matching information includes the moving position information of the reference feature; based on the second matching information, feature matching is performed on the first scene images other than the first first scene image and the marker to obtain third matching information, and the third matching information includes the matching degree between the first scene image and the marker.

[0025] In a possible implementation, the virtual reality special effects information includes model parameters of the virtual model, and the determination module is further used to obtain parameters of the camera device based on the successfully matched scene image and the marker; determine a first coordinate system corresponding to the marker using the parameters of the camera device, and determine a second coordinate system corresponding to the virtual model using the model parameters of the virtual model; obtain parameter information of the first coordinate system and the second coordinate system, and process at least one of the parameters of the camera device or the model parameters based on the parameter information to obtain rendering parameters.

[0026] In a possible implementation, the determination module is used to determine the parameter matrix corresponding to the parameters of the camera device and the model parameters when the directions of the coordinate axes of the first coordinate system and the second coordinate system are the same, and obtain the rendering parameters based on the parameter matrix; when the directions of the coordinate axes of the first coordinate system and the second coordinate system are different, determine the conversion parameters of the first coordinate system and the second coordinate system, adjust the parameters of the camera device or at least one of the model parameters based on the conversion parameters, determine the parameter matrix corresponding to the adjusted parameters of the camera device or at least one of the model parameters, and obtain the rendering parameters based on the parameter matrix.

[0027] In a possible implementation, the determination module is used to superimpose the rendering image with the scene to obtain a preview image; obtain a control instruction, the control instruction is used to confirm the preview image; and use the preview image as the special effect image according to the control instruction.

[0028] In a possible implementation, the determination module is used to detect at least one of the first trigger information or the second trigger information, and when at least one of the first trigger information or the second trigger information is detected, generate the corresponding control instruction, the first trigger information is generated based on the control of the preview interface, the preview interface includes the preview screen, and the second trigger information is generated based on at least one of the user's hand posture or hand motion trajectory.

[0029] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to enable the computer device to implement any of the special effect screen display methods described above.

[0030] On the other hand, a computer-readable storage medium is also provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor so that a computer implements any of the above-mentioned methods for displaying special effect images.

[0031] On the other hand, a computer program or a computer program product is also provided, wherein at least one computer instruction is stored in the computer program or the computer program product, and the at least one computer instruction is loaded and executed by a processor so that the computer implements any of the above-mentioned special effect screen display methods.

[0032] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0033] The technical solution provided in the embodiment of the present application utilizes the markers contained in the scene image to associate with the virtual reality special effects information, thereby improving the convenience of marker selection, and utilizes the markers in the scene image to associate with the virtual reality special effects information, and the display angle of the virtual reality special effects information is adapted to the shooting angle of the camera device, so as to present a more realistic special effects picture, and is applied to various scenes that require special effects pictures, such as in generating video tools containing special effects pictures, to improve the user interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 It is a schematic diagram of an implementation environment of a method for displaying a special effect picture provided by an embodiment of the present application;

[0036] Figure 2 It is a flow chart of a method for displaying a special effect picture provided by an embodiment of the present application;

[0037] Figure 3 is a flow chart of a method for performing feature matching on a scene image provided by an embodiment of the present application;

[0038] Figure 4 is a schematic diagram of obtaining camera extrinsic parameters provided by an embodiment of the present application;

[0039] Figure 5 This is a flow chart of a method for obtaining a special effect picture provided by an embodiment of the present application;

[0040] Figure 6 is a schematic diagram of a first coordinate system provided in an embodiment of the present application;

[0041] Figure 7 is a schematic diagram of a second coordinate system provided in an embodiment of the present application;

[0042] Figure 8 This is a schematic diagram of a display process of a special effect screen provided in an embodiment of the present application;

[0043] Fig. 9 It is a structural schematic diagram of a display device for special effect images provided in an embodiment of the present application;

[0044] Fig.10 It is a structural diagram of a terminal device provided in an embodiment of the present application;

[0045] Fig.11 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0047] Figure 1 Schematic diagram of the implementation environment of a method for displaying a special effect screen provided in an embodiment of the present application. Figure 1 As shown, the implementation environment includes: a terminal device 101 and a server 102 .

[0048] The display method of the special effect screen provided in the embodiment of the present application can be executed by the terminal device 101, or it can be jointly executed by the terminal device 101 and the server 102, and the embodiment of the present application does not limit this. In the case where the display method of the special effect screen provided in the embodiment of the present application is jointly executed by the terminal device 101 and the server 102, the server 102 undertakes the main computing work, and the terminal device 101 undertakes the secondary computing work; or, the server 102 bears the secondary computing work, and the terminal device 101 undertakes the main computing work; or, the server 102 and the terminal device 101 adopt a distributed computing architecture for collaborative computing.

[0049] Optionally, the terminal device 101 can be any electronic product that can perform human-computer interaction with a user through one or more methods such as a keyboard, a touchpad, a touch screen, a remote control, voice interaction, or a handwriting device. The terminal device 101 includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, and a vehicle-mounted terminal. In an embodiment of the present application, the terminal device 101 may have a camera device and a display screen. Based on this method, a special effect picture can be further obtained by shooting a picture with a camera device, and the special effect picture can be displayed on a display screen. The server 102 is a server, or a server cluster composed of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiment of the present application. The server 102 is connected to the terminal device 101 through a wired network or a wireless network. The server 102 has a data receiving function, a data processing function, and a data sending function. Of course, the server 102 may also have other functions, which are not limited in the embodiment of the present application.

[0050] Those skilled in the art should understand that the above-mentioned terminal device 101 and server 102 are only for illustration, and other existing or future terminal devices or servers, if applicable to the present application, should also be included in the scope of protection of the present application and are included here by reference.

[0051] The embodiment of the present application provides a method for displaying a special effect picture, which can be applied to the above implementation environment. Figure 2 Taking the flowchart of a method for displaying a special effect screen provided by an embodiment of the present application as an example, the method can be executed by an electronic device, and the electronic device can be Figure 1 The terminal device 101 in Figure 1 The server 102 and the terminal device 101 are collectively referred to in the embodiment of the present application, and this is not limited to this. Figure 2 As shown, the method includes the following steps 110 to 130.

[0052] In step 110, a scene image captured by a camera device is displayed.

[0053] In an exemplary embodiment of the present application, the scene image is a realistic picture of a real scene, which can be directly captured by a camera device, and the camera device can be a camera built into the terminal device, etc. For example, the scene image can be an image captured by a camera built into the terminal device. The terminal device turns on the camera of the camera and shoots the markers contained in the current scene in real time. During the shooting process, the markers can be shot at different angles to obtain a scene image containing the markers. The markers can be used to carry the virtual models contained in the virtual reality special effects information, and the markers are objects with texture features. Taking the special effects picture display system on the terminal device as an example, the camera device uploads the scene image containing the markers to the special effects picture display system of the terminal device, and the special effects picture display system can display the scene image.

[0054] Exemplarily, the embodiments of the present application do not limit the format of the scene image. For example, the format of the scene image may include one or more of jpg or jpeg (Joint Photographic Experts Group), png (Portable Network Graphics), tiff (Tag Image File Format), gif (Graphics Interchange Format), bmp (Bitmap), and pcx (PC Paintbrush Exchange).

[0055] In step 120, a marker selected in the scene image is determined, the marker has associated virtual reality special effect information, and when the shooting angle of the camera device changes, the association between the marker and the virtual reality special effect information remains unchanged.

[0056] In an exemplary embodiment of the present application, the marker may be a portion having texture features in a scene image. After the scene image is displayed, the user may select a marker from the scene image. The embodiment of the present application does not limit the method of selecting a marker in the scene image. Exemplarily, based on the selection operation, the special effects screen display system determines the selected marker in the scene image. For example, when displaying a scene image, selection items may be displayed for each marker included in the scene image. After the selection item of any marker is selected, the marker corresponding to the selected selection item is used as the selected marker. In order to achieve special effect display, the marker has associated virtual reality special effect information. The embodiment of the present application does not limit the association method of the marker and the virtual reality special effect information. For example, the two can be associated through the same position information, or bound based on the same identifier. In addition, since the method can be applied to a scene captured in real time by a camera device, the shooting angle of the camera device can change. When the shooting angle of the camera device changes, the association method of the marker and the virtual reality special effect information remains unchanged, thereby achieving that the virtual reality special effect information can adapt to the marker following the shooting angle of the camera device.

[0057] In one possible implementation, the process of determining virtual reality special effects information associated with a marker may include but is not limited to: displaying multiple model resources uploaded by a user, determining a selected model resource from the multiple model resources as a virtual model associated with the marker; performing special effects processing on the virtual model to obtain virtual reality special effects information for rendering the virtual model, and associating the virtual reality special effects information with the marker.

[0058] Exemplarily, the virtual model can be a virtual model uploaded by a user to the special effects screen display system, or it can be a virtual model provided by the special effects screen display system. The virtual model has corresponding model parameters, wherein the virtual model can include a two-dimensional (2D) virtual model and a three-dimensional (3D) virtual model. A two-dimensional virtual model can include but is not limited to pictures, videos, 2D text, etc., and a three-dimensional virtual model can include but is not limited to a static three-dimensional virtual model, a dynamic three-dimensional virtual model, 3D text, etc. The model parameters may include a model coordinate system established based on the virtual model and the relevant coordinates of the virtual model. Among them, the origin of the model coordinate system can be the center point of the virtual model, or it can be other points set by the user.

[0059] In an exemplary embodiment of the present application, after the virtual model is obtained, model parameters of the virtual model may also be acquired, and dimensional information of the virtual model may be determined based on the model parameters.

[0060] Exemplarily, when the dimension information indicates that the virtual model is a two-dimensional virtual model, the virtual model is decoded and converted to obtain the first virtual reality special effect information. The format of the two-dimensional virtual model may include mp4 (Moving Picture Experts Group Audio Layer IV), flv (Flash Video), avi (Audio Video Interleaved), pg (Progressive Graphics), png (Portable Network Graphics), jpeg, tiff, gif, bmp, pcx and other formats.

[0061] In an exemplary embodiment of the present application, the process of decoding and format converting the virtual model to obtain the first virtual reality special effect information may include: using a decoder to decode the virtual model to obtain a frame image in a first image format, where the first image format is an image format output by the decoder; format converting the frame image in the first image format to obtain the first virtual reality special effect information.

[0062] Taking a video file in mp4 format with a two-dimensional virtual model as an example, a decoder is used to decode the video file to obtain a plurality of frame images in a first image format, wherein the first image format is an image format output by the decoder. For example, the first image format may be in YUV format, where Y represents the brightness information of the frame image, and U and V represent the chrominance information of the frame image. The frame image in the first image format is then format-converted to obtain first virtual reality special effect information, wherein the first virtual reality special effect information can be directly used for rendering by a renderer. For example, the first virtual reality special effect information may include an image in RGB format composed of three primary colors, and the image in RGB format can be directly applied to a subsequent rendering process.

[0063] Exemplarily, when the dimension information indicates that the virtual model is a three-dimensional virtual model, the format information of the three-dimensional virtual model is determined based on the model parameters, and the virtual model is processed based on the format information to obtain the second virtual reality special effect information.

[0064] Since there are many types of 3D virtual models, in addition to the 3D virtual models in various common formats generated by common 3D modeling software, there are also some 3D virtual models in uncommon formats. For example, the common formats of 3D virtual models may include fbx (Autodesk FBX), gltf (GL Transmission Format), glb (GL Transmission Format Binary), obj (WavefrontOBJect), dae (Digital Asset Exchange), etc.

[0065] Exemplarily, after obtaining the format information of the three-dimensional virtual model, the virtual model may be processed based on the format information of the three-dimensional virtual model to obtain the second virtual reality special effect information. Processing the virtual model based on the format information to obtain the second virtual reality special effect information may include steps 121 to 123.

[0066] In step 121, based on the format information of the three-dimensional virtual model and the standard format information, a first virtual model or a second virtual model is determined, wherein the format information of the first virtual model is the same as the standard format information, and the format information of the second virtual model is different from the standard format information.

[0067] Exemplarily, the standard format information may be a commonly used format for three-dimensional virtual models, that is, a format that can be directly processed by the special effect screen display system. For example, the standard format information may include one or more of fbx, gltf, glb, obj, and dae.

[0068] In step 122, when the three-dimensional virtual model is the first virtual model, the first virtual model is parsed to obtain second virtual reality special effect information or parsing error information corresponding to the first virtual model; when parsing error information is obtained, the first virtual model corresponding to the parsing error information is format-converted to obtain second virtual reality special effect information.

[0069] In an exemplary embodiment of the present application, for a first virtual model in a standard format, the first virtual model is parsed, wherein the first virtual model may include a three-dimensional virtual model and model parameters corresponding to the three-dimensional virtual model. Due to the complexity of the three-dimensional virtual model in the modeling process, for example, the three-dimensional virtual model may be a simple virtual model with a small number of facets, or a complex virtual model with a large number of facets, texture maps, multiple materials, and multiple animation effects. There are large differences in the model parsing process for simple virtual models and complex virtual models. Therefore, for virtual models of the same format, the parsing results may be different. For the first virtual model, if the first virtual model is parsed successfully, the second virtual reality special effects information is obtained; if the first virtual model fails to be parsed, a parsing error message is obtained.

[0070] Exemplarily, for the first virtual model that fails to parse, that is, the first virtual model that obtains parsing error information, the format of the first virtual model can be converted into a custom format using a format conversion tool. During the format conversion process, the data of the format conversion process is retained, and the data of the format conversion process is detected to determine whether there is abnormal data. If there is abnormal data in the format conversion, the abnormal data is corrected. Then the first virtual model is parsed to obtain the second virtual reality special effect information. Among them, the second virtual reality special effect information may include mesh data, texture mapping, animation data, material data, etc. of the three-dimensional virtual model, and the second virtual reality special effect information can be directly used for rendering by the renderer.

[0071] In step 123, the second virtual model is converted into a format to obtain second virtual reality special effect information.

[0072] The process of format conversion of the second virtual model is similar to the process of format conversion of the first virtual seat model that failed to parse, and will not be elaborated on here. According to an exemplary embodiment of the present application, the two-dimensional virtual model and the three-dimensional virtual model are processed differently by the dimension of the virtual model to obtain the first virtual reality special effects information or the second virtual reality special effects information, which can be compatible with virtual models of more formats, provide users with more choices of virtual models, present more realistic visual effects during the rendering process, improve the user's interactive experience, and by improving the compatibility of the virtual model, the cost of model creation and maintenance can be reduced to a certain extent.

[0073] In an exemplary embodiment of the present application, after obtaining the scene image, the parameters of the camera device can also be determined using the scene image and the marker. The present application takes the marker image as an example, and the process of determining the parameters of the camera device can include: performing feature matching between the scene image and the marker; and determining the parameters of the camera device based on the successfully matched scene image and the marker.

[0074] Exemplarily, a camera device continuously captures a scene including a marked image at different angles to obtain a plurality of continuous scene images. Figure 3 This is a flow chart of performing feature matching on a scene image provided by an embodiment of the present application. Figure 3 As shown, performing feature matching on a scene image may include steps 124 to 127 .

[0075] In step 124, a feature matching interval is obtained, and a plurality of first scene images and a plurality of second scene images are determined in the plurality of scene images based on the feature matching interval, wherein the first scene images are determined based on the feature matching interval, and the second scene images are located between the first scene images.

[0076] Exemplarily, the feature matching interval may be a preset time or a preset number of scene images. The present application takes the feature matching interval as the preset number of scene images as an example. The first scene image and the second scene image in the scene image are determined by the preset number of scene images, wherein the first scene image is determined based on the feature matching interval, that is, the number of scene images between adjacent first scene images is the same as the preset number of scene images, and the second scene image is the scene image between adjacent first scene images.

[0077] In step 125, based on the generation order of the scene images, feature matching is performed on the first scene image and the marker to obtain first matching information, which includes the matching degree between the first scene image and the marker and the position information of the reference feature.

[0078] Exemplarily, among the multiple scene images, feature extraction is performed on the first first scene image to obtain key features and marker features of the first first scene image. The method of extracting features from the first first scene image includes but is not limited to extracting features from the first first scene image using a convolutional neural network (CNN). Then, the key features are used to generate corresponding feature vectors, wherein the feature vectors contain key information of the image, such as shape, texture, etc.

[0079] In a possible implementation, after feature extraction is completed, the extracted key features and marker features can also be feature matched. For example, the feature vectors corresponding to the key features and marker features in the first first scene image can be determined, and the feature vectors are vectors corresponding to the feature points in the first first scene image. The distance between the feature vectors corresponding to the key features and the marker features is calculated, wherein the distance between the feature vectors includes but is not limited to Euclidean distance, Hamming distance, cosine distance, etc. The smaller the distance between the feature vectors, the more similar the two feature points are. The first matching information of the first first scene image and the marker image is determined based on the distance between the feature vectors, wherein the first matching information may include the degree of matching and the position information of the reference feature. The reference feature may be a feature corresponding to a feature vector whose distance between the feature vector corresponding to the key feature and the feature vector corresponding to the marker feature is less than a distance threshold, and the position information of the reference feature may be the position information of the pixel block corresponding to the feature point of the reference feature, and the pixel block corresponding to the reference feature is used to track and detect the position of the marker image relative to the camera device.

[0080] In step 126, based on the first matching information, the position of the camera device corresponding to the marker and the second scene image is tracked and detected to obtain second matching information, which includes the movement position information of the reference feature.

[0081] Exemplarily, by using the position information of the pixel block where the reference feature is located contained in the first matching information, by searching the position of the pixel block corresponding to the reference feature in the second scene image, the offset of the pixel block corresponding to the reference feature can be determined, and the position of the camera device corresponding to the second scene image is tracked and detected by the offset of the pixel block corresponding to the reference feature to obtain the second matching information. The second matching information includes the moving position information of the marker image relative to the camera device, and the moving position information may include the moving direction and the moving distance.

[0082] In step 127, based on the second matching information, feature matching is performed between the first scene images except the first first scene image and the marker to obtain third matching information, where the third matching information includes the degree of matching between the first scene image and the marker.

[0083] Exemplarily, the moving position information in the second matching information can be used for auxiliary positioning in the first scene images other than the first first scene image, and the feature extraction areas of the remaining first scene images can be quickly determined to assist the remaining first scene images in feature matching with the marker image. For example, the moving position information can include the moving direction of the pixel block and the moving distance of the pixel block in the moving direction. Based on the moving position information, the moving direction and distance of the feature points of the key features and the feature points of the marker features can be determined, and the feature extraction area in the first scene image can be quickly determined.

[0084] In an exemplary embodiment of the present application, a scene image is preliminarily screened based on matching information, for example, the matching degree in the first matching information and the third matching information, to obtain a successfully matched scene image, wherein the matching degree between the successfully matched scene image and the marked image is greater than or equal to a reference matching threshold, and the reference matching threshold can be set based on actual conditions. By performing feature matching on the scene image and processing the matched scene image and the marked image in a subsequent process, the amount of calculation in the subsequent image processing process can be reduced to a certain extent, and the accuracy of the calculation can be improved.

[0085] In an exemplary embodiment of the present application, after completing the feature matching of the scene image and the marker image, the successfully matched scene image and marker image may be processed to obtain parameters of the camera device.

[0086] Exemplarily, multiple key points are determined in the marker and the matched scene image (first scene image), and the key points correspond one to one in the marker and the matched scene image. The two-dimensional coordinates of the key points of the marker in the world coordinate system and the three-dimensional coordinates of the key points of the matched scene image in the camera coordinate system are obtained, and the parameters of the camera device are determined using the two-dimensional coordinates of each key point and the corresponding three-dimensional coordinates, wherein the parameters of the camera device may include the relative position information of the marker and the camera device. For the second scene image, the position information of the marker relative to the camera device can be tracked and detected based on the moving position information in the second matching information, for example, using the moving direction of the marker in the moving position information and the moving distance in the moving direction, to obtain the parameters of the camera device. For example, when the camera device is a camera, the camera intrinsic parameters and camera extrinsic parameters can be obtained.

[0087] For example, camera intrinsic parameters include but are not limited to focal length, principal point position, and pixel size. Camera extrinsic parameters refer to the position and posture information of the camera in the world coordinate system, including but not limited to the rotation matrix and translation matrix corresponding to the camera. Camera intrinsic parameters can be obtained through camera parameters, and camera extrinsic parameters can be obtained using the PnP (Perspective-n-Points, plug-and-play) algorithm. Figure 4 FIG. 1 is a schematic diagram of obtaining camera external parameters provided by an embodiment of the present application. Figure 4 As shown, multiple reference points Pi are selected in the world coordinate system (OwXwYwZw), and the coordinates pi of the reference points in the corresponding pixel coordinate system (OcXcYcZc) are obtained. For example, n reference points P1, P2...Pn are selected, and the corresponding coordinates in the pixel coordinate system are p1, p2...pn, and the external parameters R and t of the camera are determined using DLT (Direct Linear Transform).

[0088] It should be noted that the method of obtaining camera intrinsic parameters and camera extrinsic parameters in this application is an exemplary description, and camera intrinsic parameters and camera extrinsic parameters can also be obtained by other methods, and this application does not impose any restrictions on this.

[0089] Exemplarily, after obtaining the parameters of the camera device, the parameters of the camera device may be processed to remove parameters with a small change in the relative position between the marker and the camera device. For example, a screening threshold for relative position information may be set, the change in the relative position between the marker and the camera device may be compared with the screening threshold, and parameters with a change less than the screening threshold may be removed. By processing the relative position change, a relatively stable output may be formed in the subsequent model rendering process.

[0090] In an exemplary embodiment of the present application, after the parameters of the camera device are determined, the parameters of the camera device and the model parameters may be processed to obtain a special effect picture. Figure 5 This is a flow chart of a method for obtaining a special effect picture provided by an embodiment of the present application. Figure 5 As shown, obtaining a rendering picture may include steps 141 to 144 .

[0091] In step 141 , a first coordinate system corresponding to the marker is determined using parameters of the camera device, and a second coordinate system corresponding to the virtual model is determined using model parameters of the virtual model.

[0092] For example, the first coordinate system can be established based on the marker. Taking the marker image as an example, the origin of the first coordinate system can be the center of the marker image or any point in the marker image set by the user, which is not limited in the present application. Figure 6 is a schematic diagram of a first coordinate system provided in an embodiment of the present application. Figure 6 As shown, a first coordinate system is established by taking the center of the marked image 510 as the origin of the coordinate system, an X-axis and a Z-axis are established with the plane where the marked image is located, and a Y-axis is established with the plane perpendicular to the marked image. For example, the first coordinate system is a right-handed coordinate system. After the first coordinate system is established, the coordinates of the camera device 520 in the first coordinate system can be determined by the parameters of the camera device, that is, the relative position of the camera device 520 and the marked image 510 can be determined. For example, the camera device 520 is located in the negative direction of the Y axis of the first coordinate system, and the camera device faces the positive direction of the Y axis.

[0093] Exemplarily, the virtual reality special effect information includes the processed virtual model and model parameters corresponding to the virtual model, and the model parameters may include a second coordinate system corresponding to the virtual model, wherein the origin of the second coordinate system may be the center of the virtual model or a custom point in the virtual model. Figure 7 is a schematic diagram of a second coordinate system provided in an embodiment of the present application. Figure 7 As shown, taking the virtual model as a three-dimensional virtual model as an example, a second coordinate system is established with the center of the bottom of the three-dimensional virtual model as the origin of the second coordinate system, the X-axis and Z-axis are established on the plane where the bottom of the three-dimensional virtual model is located, and the Y-axis is established on the plane perpendicular to the X-axis and Z-axis and corresponding to the extension direction of the three-dimensional virtual model.

[0094] In step 142, parameter information of the first coordinate system and the second coordinate system is obtained, and at least one of the parameters of the camera device or the model parameters is processed based on the parameter information to obtain rendering parameters.

[0095] Exemplarily, after establishing the first coordinate system and the second coordinate system, parameter information of the first coordinate system and the second coordinate system can be obtained, wherein the parameter information can include the orientation of the coordinate axis of the coordinate system and the position of the origin of the coordinate axis. The first coordinate system is compared with the second coordinate system. If the first coordinate system is the same as the second coordinate system, the parameter matrix corresponding to the parameters of the camera device and the model parameters is determined. Among them, the first coordinate system is the same as the second coordinate system in that the directions of the coordinate axes of the first coordinate system and the second coordinate system are the same. Taking the camera device as a camera as an example, the parameter matrix may include a model matrix, a view matrix, a projection matrix, and a translation matrix. The parameter matrix corresponding to the model parameters may be a model matrix. The first coordinate system may coincide with the second coordinate system through the translation matrix. The parameter matrix corresponding to the camera extrinsic parameters may be a view matrix. The parameter matrix corresponding to the camera intrinsic parameters may be a projection matrix. The view matrix is ​​used to characterize the placement position of the virtual model under the camera's perspective, and the projection matrix is ​​used to characterize the perspective relationship of the placement position of the virtual model under the camera's perspective.

[0096] For example, after obtaining the model matrix, view matrix, projection matrix and translation matrix, the product of the model matrix, view matrix, projection matrix and translation matrix can be obtained, and the product is used as the rendering parameter of the virtual model. After adjusting the rendering parameters of the virtual model, the virtual model can satisfy the perspective relationship under the camera perspective, and the virtual model is located at the specified position of the marked image and the orientation meets the requirements.

[0097] Exemplarily, if the first coordinate system is different from the second coordinate system, the conversion parameters of the first coordinate system and the second coordinate system are determined. The difference between the first coordinate system and the second coordinate system is that the direction of at least one coordinate axis in the first coordinate system and the second coordinate system is different; the conversion parameters of the first coordinate system and the second coordinate system may include the rotation angle and translation parameters of the coordinate system. Figure 6 and Figure 7 For example, you can Figure 6 After the first coordinate system is rotated 180° clockwise, we get Figure 7 The second coordinate system in the same direction as the coordinate system, and then Figure 6 The first coordinate system in is translated based on the translation parameter, and the result is Figure 7 The same coordinate system, at this time, the origin of the first coordinate system coincides with the origin of the second coordinate system, and the corresponding coordinate axes are in the same direction.

[0098] Exemplarily, the conversion parameter may be one matrix or two matrices. When the conversion parameter is two matrices, it may include a rotation matrix and a translation matrix, wherein a 3*3 rotation matrix may be used to adjust the rotation angle of the coordinate axis in the coordinate system, and a 1*3 translation matrix may be used to control the translation of the coordinate system; when the conversion parameter is one matrix, a 3*3 rotation matrix and a 1*3 translation matrix may be combined to form a 3*4 matrix, or the 4th row of data in the 3*4 matrix may be padded to form a 4*4 square matrix, and the rotation and translation of the coordinate system may be simultaneously realized through a 3*4 matrix or a 4*4 square matrix.

[0099] Then, based on the information such as the conversion parameter, the placement position of the virtual model in the marked image, and the orientation of the virtual model, at least one of the model matrix, the view matrix, and the projection matrix is ​​adjusted, and the product of the adjusted model matrix, the view matrix, and the projection matrix is ​​obtained, and the product of the adjusted model matrix, the view matrix, and the projection matrix is ​​used as the rendering parameter. After the rendering parameters of the virtual model are adjusted, the virtual model can satisfy the perspective relationship under the camera viewing angle, and the virtual model is located at a specified position of the marked image and the orientation meets the requirements.

[0100] In the exemplary embodiment of the present application, the number of virtual models may be one or more. When the number of virtual models is multiple, the multiple virtual models may be located in the same coordinate system or in different coordinate systems, and the dimensions of the multiple virtual models may be the same or different. When the virtual models are located in the same coordinate system, i.e., the second coordinate system, the position and posture of the multiple virtual models relative to the marker may be adjusted by adjusting the view matrix, the projection matrix, and the model matrix, or the position and posture of each virtual model may be adjusted by adjusting the model matrix of a single virtual model, and the present application does not limit this.

[0101] In step 143 , the marker and the virtual model are rendered based on the rendering parameters to obtain a rendering image.

[0102] Exemplarily, the rendering parameters may include but are not limited to texture maps, animation data, material data, etc. of the virtual model and the marker, and the virtual model and the marker are rendered based on the rendering parameters to obtain a rendered image.

[0103] In step 144, the rendering image is superimposed on the scene image to obtain a special effect image.

[0104] In an exemplary embodiment of the present application, the format of the scene image can also be determined. If the format of the scene image satisfies the overlay format, the rendering picture is overlaid with the scene image; if the format of the scene image does not satisfy the overlay format, the format of the scene image is converted, and the scene image after the format conversion is overlaid with the rendering picture. Among them, the overlay format can be a format that can be processed by the special effect picture display system. For example, the special effect picture display system can process formats such as jpg, jpeg, png, tiff, gif, bmp, and pcx. If the format of the scene image is other formats, the format of the scene image can be converted into a format that can be processed by the special effect picture display system.

[0105] Exemplarily, the process of superimposing the rendering screen with the scene image to obtain a special effects screen may include: superimposing the rendering screen with the scene image to obtain a preview screen; obtaining a control instruction, the control instruction being used to confirm the preview screen; and using the preview screen as a special effects screen according to the control instruction.

[0106] For example, the preview interface may include a preview screen and controls, and the preview screen may be an overlay screen of the rendering screen and the scene image.

[0107] The process of obtaining control instructions may include: detecting at least one of the first trigger information or the second trigger information, and generating a corresponding control instruction when at least one of the first trigger information or the second trigger information is detected, the first trigger information is generated based on the control of the preview interface, the preview interface includes a preview screen, and the second trigger information is generated based on at least one of the user's hand posture or hand motion trajectory.

[0108] Exemplarily, the first trigger information corresponding to the control is detected. The control may include but is not limited to a zoom-in control, a zoom-out control, a move control, a rotation control, and a confirmation control. When a trigger operation of any of the above controls is detected, the corresponding first trigger information is generated, and a corresponding control instruction is generated based on the first trigger information. The control instruction may include but is not limited to a zoom-in, zoom-out, move, rotation instruction and a confirmation instruction of the preview screen.

[0109] For example, the preview interface can also detect at least one of the user's hand posture and hand motion trajectory. When it is detected that the user's hand posture satisfies at least one of the specified actions or hand motion trajectories, second trigger information can be generated based on at least one of the user's hand posture or hand motion trajectory, and corresponding control instructions can be generated based on the second trigger information. The control instructions may include but are not limited to instructions for zooming in, zooming out, moving, rotating, and confirming the preview screen.

[0110] Exemplarily, the present application can also control the preview screen to perform corresponding operations through control instructions. After the user confirms the preview screen, a confirmation instruction can be obtained. After receiving the confirmation instruction, the preview screen is used as a special effects screen. When the generated preview screen does not meet the user's requirements, the rendering parameters can be adjusted, and the corresponding preview screen is obtained based on the adjusted rendering parameters. The preview screen is confirmed again until the preview screen meets the user's requirements and a confirmation instruction is obtained, and the preview screen is used as a special effects screen.

[0111] In step 130, a special effect picture is displayed, where the special effect picture includes virtual reality special effect information associated with a marker displayed on the scene image, and the display angle of the virtual reality special effect information is adapted to the shooting angle of the camera device.

[0112] In an exemplary embodiment of the present application, the generated special effects picture can be displayed with a transmission frame rate greater than or equal to a preset frame rate per second, wherein the special effects picture includes virtual reality special effects information associated with a marker displayed on the scene image. For example, the preset frame rate can be 30, and the smooth playback of the special effects picture can be guaranteed by setting the transmission frame rate of the special effects picture per unit time. In addition, since the scene picture is obtained by shooting a shooting device, the display angle of the virtual reality special effects information is adapted to the shooting angle of the camera device, and the user can follow the shooting angle of the shooting device to watch the virtual reality special effects information. In this case, the user can directly enhance the reality technology (Augmented Reality, AR) effect in real time through the screen of the terminal device, and can also observe and preview the virtual 3D model, that is, the virtual reality special effects information associated with the marker, from any angle or position on the premise of ensuring that the marker is always in the original scene picture.

[0113] For example, the markers in the AR scene are captured by the camera of the terminal device, and the position and direction of displaying the virtual reality special effects information are determined according to the markers; various 2D / 3D special effects are generated according to the preset special effects generation algorithm; finally, the special effects are integrated with the AR scene to obtain the special effects picture, and the fused AR scene, that is, the special effects picture, is displayed to the user, thereby improving the realism and fidelity of the displayed content and achieving the effect of enhancing the user's interactive experience.

[0114] In a possible implementation, the special effects screen also includes a processing control. The embodiment of the present application does not limit the type and number of the processing control, and the processing control can be used to process the special effects screen. For example, the processing control includes at least one of a sharing control, a publishing control, or a modification control, wherein the sharing control is used to share the special effects screen, the publishing control is used to publish the special effects screen, and the modification control is used to modify the special effects screen. Regardless of the type of processing control, after displaying the special effects screen, it also includes: when the processing control is triggered, the special effects screen is subjected to a processing operation corresponding to the processing control. For example, when the special effects screen includes a publishing control, the special effects screen is subjected to a publishing operation.

[0115] In addition, the method can be applied to various scenarios that require special effects images. The embodiment of the present application does not limit the application scenarios of the method. For example, the method can be applied to a video tool that generates special effects images. By applying the method on the video tool, the user can generate special effects images that meet the user's needs, and support users to use the matching video tool to generate and preview custom AR special effects scene images, and the preview effect is strictly aligned with the final display effect, which can achieve the "what you see is what you get" effect. In addition, after obtaining the special effects image, operations such as publishing and sharing can be performed to improve the user's interactive experience.

[0116] According to an exemplary embodiment of the present application, a scene image is obtained based on the camera device, and the scene image may contain a marker. The marker may include a specific object in the scene, a marker image uploaded by the user, and a marker image provided by the special effect screen display system, thereby improving the convenience of marker selection. For a virtual model associated with a marker, the virtual model is processed to obtain virtual reality special effect information of the virtual model, which can be compatible with virtual models in more formats, providing users with more choices. Thus, a more realistic special effect screen can be presented during the rendering process. This method can be applied to various scenes that require special effect screens, such as being applied to a video tool that generates special effect screens, to improve the user's interactive experience.

[0117] Figure 8 This is a flow chart of displaying a special effect screen provided by an embodiment of the present application. Figure 8As shown, the user inputs the marker, virtual model and scene image into the special effect picture display system, and the special effect picture display system converts the scene image of the YUV format image into an RGB format image, generates the scene image using the RGB format image and the marker, and calculates the parameters of the camera device using the scene image. The parameters of the camera device may include camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters, camera extrinsic parameters and RGB format images are used to perform special effect rendering on the virtual model to obtain a special effect picture. It should be noted that after obtaining the marker and virtual model uploaded by the user book, the present application may optionally process the marker and the virtual model, for example, format conversion processing, such as Figure 8 The steps are shown in dashed lines.

[0118] It should be noted that Figure 8 The various steps of displaying the special effects screen have been specifically described in steps 110 to 130, and will not be described in detail here.

[0119] The present application also provides a display device for special effect images. Fig. 9 is a block diagram of a display device for a special effect picture provided by an embodiment of the present application. Fig. 9 As shown, the display device of the special effect picture may include:

[0120] The first display module 210 is used to display the scene image captured by the camera device.

[0121] The determination module 220 is used to determine the selected marker in the scene image, the marker has associated virtual reality special effect information, and when the shooting angle of the camera device changes, the association between the marker and the virtual reality special effect information remains unchanged.

[0122] The second display module 230 is used to display a special effect picture, which includes virtual reality special effect information associated with a marker displayed on the scene image, and the display angle of the virtual reality special effect information is adapted to the shooting angle of the camera device.

[0123] In one possible implementation, the special effects picture also includes a processing control, which includes at least one of a sharing control, a publishing control, or a modification control. The sharing control is used to share the special effects picture, the publishing control is used to publish the special effects picture, and the modification control is used to modify the special effects picture; the second display module 230 is also used to perform a processing operation corresponding to the processing control on the special effects picture when the processing control is triggered.

[0124] In one possible implementation, the determination module 220 is used to display multiple model resources uploaded by the user, determine a selected model resource from the multiple model resources as a virtual model associated with the marker; perform special effects processing on the virtual model to obtain virtual reality special effects information for rendering the virtual model, and associate the virtual reality special effects information with the marker.

[0125] In a possible implementation, the virtual reality special effect information includes first virtual reality special effect information and second virtual reality special effect information. The determination module 220 is used to obtain model parameters of the virtual model, and determine dimension information of the virtual model based on the model parameters; when the dimension information indicates that the virtual model is a two-dimensional virtual model, the virtual model is decoded and format converted to obtain the first virtual reality special effect information; when the dimension information indicates that the virtual model is a three-dimensional virtual model, the format information of the three-dimensional virtual model is determined based on the model parameters, and the virtual model is processed based on the format information to obtain the second virtual reality special effect information.

[0126] In a possible implementation, the determination module 220 is used to use a decoder to decode the virtual model to obtain a frame image in a first image format, where the first image format is an image format output by the decoder; and to perform format conversion on the frame image in the first image format to obtain first virtual reality special effect information.

[0127] In a possible implementation, the determination module 220 is used to determine the first virtual model or the second virtual model based on the format information and the standard format information of the three-dimensional virtual model, the format information of the first virtual model is the same as the standard format information, and the format information of the second virtual model is different from the standard format information; when the three-dimensional virtual model is the first virtual model, the first virtual model is parsed to obtain second virtual reality special effect information or parsing error information corresponding to the first virtual model; when the parsing error information is obtained, the first virtual model corresponding to the parsing error information is format converted to obtain second virtual reality special effect information; when the three-dimensional virtual model is the second virtual model, the second virtual model is format converted to obtain second virtual reality special effect information.

[0128] In a possible implementation, the determination module 220 is also used to perform feature matching on multiple scene images and markers; obtain rendering parameters based on the successfully matched scene images and markers; render the markers and virtual models based on the rendering parameters to obtain a rendered picture; and superimpose the rendered picture with the scene image to obtain a special effects picture.

[0129] In one possible implementation, the determination module 220 is used to obtain a feature matching interval, determine multiple first scene images and multiple second scene images in multiple scene images based on the feature matching interval, the first scene image is determined based on the feature matching interval, and the second scene image is located between the first scene images; based on the generation order of the scene images, feature-match the first first scene image with the marker to obtain first matching information, the first matching information includes the matching degree between the first first scene image and the marker and the position information of the reference feature; based on the first matching information, track and detect the position of the camera device corresponding to the marker and the second scene image to obtain second matching information, the second matching information includes the moving position information of the reference feature; based on the second matching information, feature-match the first scene images other than the first first scene image with the marker to obtain third matching information, the third matching information includes the matching degree between the first scene image and the marker.

[0130] In one possible implementation, the virtual reality special effects information includes model parameters of the virtual model. The determination module 220 is also used to obtain parameters of the camera device based on the successfully matched scene image and marker; use the parameters of the camera device to determine the first coordinate system corresponding to the marker, and use the model parameters of the virtual model to determine the second coordinate system corresponding to the virtual model; obtain parameter information of the first coordinate system and the second coordinate system, and process at least one of the parameters of the camera device or the model parameters based on the parameter information to obtain rendering parameters.

[0131] In one possible implementation, the determination module 220 is used to determine the parameter matrix corresponding to the parameters and model parameters of the camera device when the directions of the coordinate axes of the first coordinate system and the second coordinate system are the same, and obtain rendering parameters based on the parameter matrix; when the directions of the coordinate axes of the first coordinate system and the second coordinate system are different, determine the conversion parameters of the first coordinate system and the second coordinate system, adjust at least one of the parameters or model parameters of the camera device based on the conversion parameters, determine the parameter matrix corresponding to at least one of the adjusted parameters or model parameters of the camera device, and obtain rendering parameters based on the parameter matrix.

[0132] In a possible implementation, the determination module 220 is used to superimpose the rendering picture with the scene image to obtain a preview picture; obtain a control instruction, the control instruction is used to confirm the preview picture; and use the preview picture as a special effect picture according to the control instruction.

[0133] In one possible implementation, the determination module 220 detects at least one of the first trigger information or the second trigger information, and generates a corresponding control instruction when at least one of the first trigger information or the second trigger information is detected, the first trigger information is generated based on the control of the preview interface, the preview interface includes a preview screen, and the second trigger information is generated based on at least one of the user's hand posture or hand motion trajectory.

[0134] According to the exemplary embodiment of the present application, by processing the virtual model, the virtual reality special effect information of the virtual model is obtained, which can be compatible with virtual models in more formats, providing more choices for users. In the rendering process, the marker and the virtual model are rendered using the virtual reality special effect information and the parameters of the camera device to obtain a special effect picture. This presents a more realistic visual effect and improves the user's interactive experience.

[0135] It should be understood that the above-mentioned device only uses the division of the above-mentioned functional modules as an example to illustrate when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0136] Fig.10 A block diagram of a terminal device provided in an embodiment of the present application. The terminal device 1100 may be a portable mobile terminal, such as a smart phone, a tablet computer, a laptop computer, or a desktop computer. The terminal device 1100 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, an intelligent voice interaction device, a vehicle-mounted terminal, or other names.

[0137] Typically, the terminal device 1100 includes: a processor 1101 and a memory 1102 .

[0138] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0139] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store at least one instruction, which is executed by the processor 1101 to implement the present application. Figure 2 The method embodiment shown provides a method for displaying a special effect picture.

[0140] In some embodiments, the terminal device 1100 may further optionally include: a peripheral device interface 1103 and at least one peripheral device. The processor 1101, the memory 1102 and the peripheral device interface 1103 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1103 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107 and a power supply 1109.

[0141] The peripheral device interface 1103 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1101 and the memory 1102. In some embodiments, the processor 1101, the memory 1102, and the peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1101, the memory 1102, and the peripheral device interface 1103 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0142] The radio frequency circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1104 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1104 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1104 can communicate with other terminal devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1104 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0143] The display screen 1105 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1105 is a touch display screen, the display screen 1105 also has the ability to collect touch signals on the surface or above the surface of the display screen 1105. The touch signal can be input to the processor 1101 as a control signal for processing. At this time, the display screen 1105 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1105 can be one, which is set on the front panel of the terminal device 1100; in other embodiments, the display screen 1105 can be at least two, which are respectively set on different surfaces of the terminal device 1100 or are folded; in other embodiments, the display screen 1105 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal device 1100. Even, the display screen 1105 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0144] The camera assembly 1106 is used to capture images or videos. Optionally, the camera assembly 1106 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal device 1100, and the rear camera is arranged on the back of the terminal device 1100. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0145] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1101 for processing, or input them into the radio frequency circuit 1104 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal device 1100. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0146] The power supply 1109 is used to power various components in the terminal device 1100. The power supply 1109 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1109 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery charged through a wired line, and a wireless rechargeable battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0147] In some embodiments, the terminal device 1100 further includes one or more sensors 1110 , including but not limited to: an acceleration sensor 1111 , a gyroscope sensor 1112 , a pressure sensor 1113 , an optical sensor 1115 , and a proximity sensor 1116 .

[0148] The acceleration sensor 1111 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal device 1100. For example, the acceleration sensor 1111 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1101 can control the display screen 1105 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1111. The acceleration sensor 1111 can also be used to collect game or user motion data.

[0149] The gyroscope sensor 1112 can detect the body direction and rotation angle of the terminal device 1100, and the gyroscope sensor 1112 can cooperate with the acceleration sensor 1111 to collect the user's 3D actions on the terminal device 1100. The processor 1101 can implement the following functions based on the data collected by the gyroscope sensor 1112: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0150] The pressure sensor 1113 can be set on the side frame of the terminal device 1100 and / or the lower layer of the display screen 1105. When the pressure sensor 1113 is set on the side frame of the terminal device 1100, the user's holding signal of the terminal device 1100 can be detected, and the processor 1101 performs left and right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 1113. When the pressure sensor 1113 is set on the lower layer of the display screen 1105, the processor 1101 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1105. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0151] The optical sensor 1115 is used to collect the ambient light intensity. In one embodiment, the processor 1101 can control the display brightness of the display screen 1105 according to the ambient light intensity collected by the optical sensor 1115. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1105 is increased; when the ambient light intensity is low, the display brightness of the display screen 1105 is reduced. In another embodiment, the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 according to the ambient light intensity collected by the optical sensor 1115.

[0152] The proximity sensor 1116, also known as a distance sensor, is usually arranged on the front panel of the terminal device 1100. The proximity sensor 1116 is used to collect the distance between the user and the front of the terminal device 1100. In one embodiment, when the proximity sensor 1116 detects that the distance between the user and the front of the terminal device 1100 is gradually decreasing, the processor 1101 controls the display screen 1105 to switch from the screen-on state to the screen-off state; when the proximity sensor 1116 detects that the distance between the user and the front of the terminal device 1100 is gradually increasing, the processor 1101 controls the display screen 1105 to switch from the screen-off state to the screen-on state.

[0153] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the terminal device 1100, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0154] Fig.111 is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1200 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Unit, CPU) 1201 and one or more memories 1202, wherein the one or more memories 1202 store at least one program code, and the at least one program code is loaded and executed by the one or more processors 1201 to implement Figure 2 Of course, the server 1200 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output, and the server 1200 may also include other components for realizing device functions, which will not be described in detail here.

[0155] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein at least one program code is stored in the storage medium, and the at least one program code is loaded and executed by a processor to enable a computer to implement the above Figure 2 The method embodiment shown provides a method for displaying a special effect picture.

[0156] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0157] In an exemplary embodiment, a computer program or a computer program product is also provided, wherein at least one computer instruction is stored in the computer program or the computer program product, and the at least one computer instruction is loaded and executed by a processor to enable a computer to implement the above Figure 2 The method embodiment shown provides a method for displaying a special effect picture.

[0158] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the virtual reality special effects information, scene images and special effects images involved in this application are all obtained with full authorization.

[0159] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0160] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for displaying a special effect picture, characterized in that: The method comprises: Displaying a scene image captured by a camera device; Determine a marker selected in the scene image, the marker having associated virtual reality special effect information, and when the shooting angle of the camera device changes, the association between the marker and the virtual reality special effect information remains unchanged; Displaying a special effect picture, wherein the special effect picture includes virtual reality special effect information associated with the marker displayed on the scene image, and the display angle of the virtual reality special effect information is adapted to the shooting angle of the camera device; Before displaying the special effects screen, the method further includes: Displaying multiple model resources uploaded by the user, and determining a selected model resource from the multiple model resources as a virtual model associated with the marker; Performing special effects processing on the virtual model to obtain virtual reality special effects information for rendering the virtual model, and associating the virtual reality special effects information with the marker; the virtual reality special effects information includes first virtual reality special effects information or second virtual reality special effects information, and performing special effects processing on the virtual model to obtain virtual reality special effects information for rendering the virtual model includes: Acquiring model parameters of the virtual model, and determining dimension information of the virtual model based on the model parameters; When the dimension information indicates that the virtual model is a two-dimensional virtual model, performing decoding processing and format conversion processing on the virtual model to obtain the first virtual reality special effect information; In a case where the dimension information indicates that the virtual model is a three-dimensional virtual model, format information of the three-dimensional virtual model is determined based on the model parameters, and the virtual model is processed based on the format information to obtain the second virtual reality special effect information; the processing of the virtual model based on the format information to obtain the second virtual reality special effect information includes: Determining a first virtual model or a second virtual model based on the format information and the standard format information of the three-dimensional virtual model, wherein the format information of the first virtual model is the same as the standard format information, and the format information of the second virtual model is different from the standard format information; When the three-dimensional virtual model is the first virtual model, the first virtual model is parsed to obtain the second virtual reality special effect information or parsing error information corresponding to the first virtual model; when the parsing error information is obtained, the format of the first virtual model corresponding to the parsing error information is converted to obtain the second virtual reality special effect information; When the three-dimensional virtual model is the second virtual model, the second virtual model is format converted to obtain the second virtual reality special effect information.

2. The method according to claim 1, characterized in that: The special effect picture further includes a processing control, and the processing control includes at least one of a sharing control, a publishing control, or a modification control, the sharing control is used to share the special effect picture, the publishing control is used to publish the special effect picture, and the modification control is used to modify the special effect picture; After displaying the special effect screen, the method further includes: When the processing control is triggered, a processing operation corresponding to the processing control is performed on the special effect picture.

3. The method according to claim 1, characterized in that The decoding and format conversion of the virtual model to obtain the first virtual reality special effect information includes: Decoding the virtual model using a decoder to obtain a frame image in a first image format, where the first image format is an image format output by the decoder; The frame image in the first image format is format converted to obtain the first virtual reality special effect information.

4. The method according to claim 1, characterized in that: After determining the selected marker in the scene image, the method further includes: Performing feature matching between a plurality of scene images and the marker; Acquiring rendering parameters according to the successfully matched scene image and the marker; Rendering the marker and the virtual model based on the rendering parameters to obtain a rendering picture; The rendering picture is superimposed on the scene image to obtain the special effect picture.

5. The method according to claim 4, characterized in that The performing feature matching between the multiple scene images and the markers includes: Acquire a feature matching interval, and determine a plurality of first scene images and a plurality of second scene images from the plurality of scene images based on the feature matching interval, wherein the first scene images are determined based on the feature matching interval, and the second scene images are located between the first scene images; Based on the generation order of the scene images, feature matching is performed on the first scene image and the marker to obtain first matching information, wherein the first matching information includes a matching degree between the first scene image and the marker and position information of a reference feature; Based on the first matching information, tracking and detecting the position of the camera device corresponding to the marker and the second scene image to obtain second matching information, wherein the second matching information includes the moving position information of the reference feature; Based on the second matching information, feature matching is performed between first scene images other than the first first scene image and the marker to obtain third matching information, wherein the third matching information includes a matching degree between the first scene image and the marker.

6. The method according to claim 4, characterized in that The virtual reality special effect information includes model parameters of the virtual model, and the obtaining of rendering parameters according to the successfully matched scene image and the marker includes: Obtaining parameters of the camera device based on the successfully matched scene image and the marker; Determine a first coordinate system corresponding to the marker by using the parameters of the camera device, and determine a second coordinate system corresponding to the virtual model by using the model parameters of the virtual model; Parameter information of the first coordinate system and the second coordinate system is acquired, and at least one of the parameters of the camera device or the model parameters is processed based on the parameter information to obtain rendering parameters.

7. The method according to claim 6, characterized in that The step of processing at least one of the parameters of the camera device or the model parameters based on the parameter information to obtain rendering parameters includes: When the directions of the coordinate axes of the first coordinate system and the second coordinate system are the same, determining a parameter matrix corresponding to the parameters of the camera device and the model parameters, and obtaining the rendering parameters based on the parameter matrix; When the directions of the coordinate axes of the first coordinate system and the second coordinate system are different, the conversion parameters of the first coordinate system and the second coordinate system are determined, and the parameters of the camera device or at least one of the model parameters are adjusted based on the conversion parameters, and a parameter matrix corresponding to at least one of the adjusted parameters of the camera device or the model parameters is determined, and the rendering parameters are obtained based on the parameter matrix.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor so that the computer device implements the method for displaying special effect pictures described in any one of claims 1-7.

9. A computer program product, characterized in that The computer program product stores at least one computer instruction, and the at least one computer instruction is loaded and executed by a processor so that the computer implements the method for displaying any one of the special effect pictures of claims 1-7.

Images