Embedding digital content in virtual space

Abstract

The present disclosure proposes a method for embedding digital content in a virtual space. The behavior of a user in the virtual space can be detected. Behavior information associated with the behavior can be obtained. Digital content can be determined based at least on the behavior information. The digital content can be added to the virtual space. The present disclosure also proposes a virtual space application device. The virtual space application device may include: a presentation unit for presenting a virtual space; a behavior recognition unit for detecting the behavior of a user in the virtual space and obtaining behavior information associated with the behavior; and a processing unit for determining digital content based at least on the behavior information and adding the digital content to the virtual space.

Description

Background Art

[0001] Virtual Reality (VR) technology integrates various technologies such as computer graphics technology, human-computer interaction technology, and sensor technology. It can use computers to generate a three-dimensional virtual space that can provide multi-sensory experiences such as vision, hearing, and touch. People can get such an experience through specific equipment such as VR devices, as if they had entered the virtual space. VR technology can be widely used in fields such as entertainment, education, military, and medicine. Taking the application of VR technology in the entertainment field as an example, VR technology can be used to reconstruct a virtual space corresponding to, for example, a concert scene, so that people can use VR equipment to get a viewing experience as if they were on the scene without having to go to the performance venue. Summary of the invention

[0002] This Summary is provided to introduce a group of concepts that will be further described in the following Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] The embodiments of the present disclosure propose a method for embedding digital content in a virtual space. The behavior of a user in the virtual space can be detected. Behavior information associated with the behavior can be obtained. Digital content can be determined based at least on the behavior information. The digital content can be added to the virtual space. The embodiments of the present disclosure also propose a virtual space application device. The virtual space application device may include: a presentation unit for presenting a virtual space; a behavior recognition unit for detecting the behavior of a user in the virtual space and obtaining behavior information associated with the behavior; and a processing unit for determining digital content based at least on the behavior information and adding the digital content to the virtual space.

[0004] It should be noted that one or more of the above aspects include the features specifically pointed out in the following detailed description and claims. The following description and drawings set forth in detail certain illustrative features of the one or more aspects. These features are merely indicative of the various ways in which the principles of various aspects may be implemented, and the present disclosure is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects.

[0006] Figure 1 An exemplary virtual space service network architecture according to an embodiment of the present disclosure is shown.

[0007] Figure 2 An exemplary process for embedding digital content in a virtual space according to an embodiment of the present disclosure is shown.

[0008] Figure 3 An exemplary scoring model according to an embodiment of the present disclosure is shown.

[0009] Figure 4 An exemplary process for training a scoring model according to an embodiment of the present disclosure is shown.

[0010] Figure 5 An exemplary terminal device according to an embodiment of the present disclosure is shown.

[0011] Figure 6 is a flowchart of an exemplary method for embedding digital content in a virtual space according to an embodiment of the present disclosure.

[0012] Figure 7 An exemplary virtual space application device according to an embodiment of the present disclosure is shown.

[0013] Figure 8 An exemplary apparatus for embedding digital content in a virtual space according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0014] The present disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that the discussion of these embodiments is only used to enable those skilled in the art to better understand and thereby implement the embodiments of the present disclosure, and does not teach any limitation on the scope of the present disclosure.

[0015] It is desirable to embed specific digital content in a virtual space so that users entering the virtual space can view the digital content and even interact with it. Currently, there are studies and developments related to embedding digital content in videos. For example, an area suitable for adding digital content can be detected from a video or created therein, suitable digital content can be determined, and the determined digital content can be added to the detected or created area in the video. This process requires a lot of computing resources and bandwidth resources.

[0016] The embodiments of the present disclosure propose a method for embedding digital content in a virtual space. In this article, a virtual space may refer to a computer-generated three-dimensional space that is different from the real environment in which the user is located, which may also be referred to as a virtual environment, a virtual scene, a virtual world, etc. For example, VR technology may be used to construct a virtual space. In this article, digital content may broadly refer to content intended to be presented with a virtual space for various purposes, such as commercial information, public service announcements, etc. Digital content may have a flexible and rich style, which may include, for example, pictures, videos, animations, etc., and may be two-dimensional or three-dimensional. Users may interact with digital content in various forms. For example, an instance of a two-dimensional digital content may be a travel video, and a user may watch the travel video in a virtual space. For example, an instance of a three-dimensional digital content may be an exhibition hall, and a user may enter the exhibition hall in a virtual space and browse and visit the exhibition hall. For example, an instance of a three-dimensional digital content may be a car, and a user may touch the car in a virtual space or even test drive it. Digital content embedding may be achieved by determining digital content based at least on the detected behavior of the user in the virtual space and adding the determined digital content to the virtual space. The digital content embedding method according to an embodiment of the present disclosure may be performed at a terminal device. In this document, a terminal device may refer to a device located locally in a user and capable of providing a virtual space experience to the user, which may also be referred to as a virtual space application device. A terminal device according to an embodiment of the present disclosure may include multiple units, which may be co-located, for example, in a single machine, or one or more of which may be separated from other units in terms of location.

[0017] In one aspect, embodiments of the present disclosure propose that digital content to be added to a virtual space can be determined based at least on the user's behavior in the virtual space. The user's behavior in the virtual space may include, for example, the user's behavior related to existing digital content in the virtual space, such as watching, walking towards, touching digital content, etc. Such behavior may indicate, for example, whether the user is currently interested in the existing digital content, etc. The user's behavior in the virtual space may also include, for example, the user's behavior related to environmental objects in the virtual space, such as walking towards the stage, waving to the singer, leaving the seat, etc. in a virtual space about a live concert. In this article, environmental objects may refer to original or inherent objects in the virtual space, such as the stage, singer, auditorium, seats, etc. in a virtual space about a live concert. Such behavior may indicate the user's current characteristics, such as personality, hobbies, habits, etc. Considering the user's behavior in the virtual space when determining the digital content to be added to the virtual space can help determine the digital content that the user is currently interested in, thereby improving the quality of the added digital content.

[0018] In another aspect, the embodiments of the present disclosure propose that the user's behavior in the virtual space can be detected by a behavior recognition unit, and the behavior recognition unit can be integrated into the terminal device together with other units. In this way, the user can switch from the real environment to the virtual space and interact with the virtual space through only a single device, which helps to obtain an immersive experience of the virtual space.

[0019] In another aspect, the embodiment of the present disclosure proposes that when determining digital content, each candidate digital content in a group of candidate digital content can be scored based on multiple indicators, and the candidate digital content with the highest score can be selected. These indicators can include, for example, the type and / or duration of the predicted behavior of the user interacting with the candidate digital content. The type of behavior can, for example, include watching, walking, touching, etc. The values ​​of these indicators of the candidate digital content can be predicted respectively by the deep learning model according to the embodiment of the present disclosure, and the candidate digital content can be scored based on the predicted values. In this way, the digital content of interest to the user can be selected more accurately and comprehensively. In addition, different weights can be set for each indicator according to actual application requirements. For example, a larger weight can be set for an indicator of greater concern, so that the digital content with better performance of the indicator can be selected.

[0020] In another aspect, the embodiment of the present disclosure proposes that a deep learning model for scoring candidate digital content is obtained by training an initial deep learning model at a group of terminal devices of a group of users. In this article, the deep learning model for scoring candidate digital content may be referred to as a scoring model. For example, the initial scoring model may be distributed to a group of terminal devices of a group of users. Each user may use his terminal device to enter a virtual space embedded with a group of digital content. The user's behavior for each digital content may be used to train the corresponding initial scoring model. The scoring models trained at each terminal device may be aggregated into an updated scoring model. The updated scoring model may be deployed for scoring candidate digital content. In this way, user behavior is only used locally by the user and will not be uploaded to the server, so that the leakage of user privacy may be avoided. In addition, the updated scoring model obtained by aggregating the trained models at each user does not have information strongly related to a specific user, so that user privacy may be further protected.

[0021] In another aspect, the embodiments of the present disclosure propose that a group of candidate digital contents from which digital contents are to be selected can be pre-received from a server and stored in a terminal device. For example, a predetermined number of digital contents can be preliminarily selected from a large amount of digital contents at the server as a group of candidate digital contents. The terminal device can receive the group of candidate digital contents from the server in advance, for example, it can be received regularly, before the virtual space is presented, when the user logs in, etc. In addition, the embodiments of the present disclosure also propose that a position suitable for adding digital contents in the virtual space can be pre-defined. For example, a group of positions suitable for adding digital contents can be pre-detected and / or created. The reconstruction of the virtual space requires real-time collection of live data and re-rendering of the virtual space based on the collected data. Taking the virtual space for a live concert as an example, the live data can include, for example, the lighting effects of the venue, the voice / posture / movement of the singer, the position / behavior of the user, etc. This process occupies more computing resources and / or bandwidth resources. The embodiments of the present disclosure provide that by pre-receiving candidate digital content and / or pre-defining the location where the digital content is to be added, it is possible to avoid excessively occupying bandwidth and / or computing resources when presenting the virtual space in real time, so that digital content can be added in a near real-time manner based on the user's behavior in the virtual space. This is particularly beneficial in the case of a virtual space based on live video.

[0022] It should be understood that although the foregoing discussion and the following discussion may involve examples of embedding digital content in a virtual space constructed using VR technology, the embodiments of the present disclosure are not limited to this, but rather digital content may be embedded in a virtual space constructed using other technologies such as augmented reality technology, mixed reality technology, etc. in a similar manner.

[0023] Figure 1 An exemplary virtual space service network architecture 100 according to an embodiment of the present disclosure is shown. The architecture 100 can present a virtual space with embedded digital content to a user 122.

[0024] The architecture 100 may include various network entities that may be interconnected directly or through a network, such as a two-dimensional content source 102 , a server 104 , a digital content provider 110 , a terminal device 112 , and the like.

[0025] The two-dimensional content source 102 may represent various network entities that can provide the server 104 with two-dimensional content resources suitable for reconstructing a virtual space. The two-dimensional content may include, for example, images, videos, etc. Images may include, for example, game screens, pictures of buildings, etc. Videos may include, for example, movies, performance videos, sports event videos, etc., which may be pre-recorded videos or live videos. It should be understood that although the above section and other sections of the present disclosure may involve adding digital content to a virtual space reconstructed based on two-dimensional content such as images, videos, etc., the embodiments of the present disclosure are not limited thereto, but digital content may be added in a virtual space obtained by other methods, such as a directly created virtual space, in a similar manner.

[0026] The digital content provider 110 may refer to various network entities that can provide digital content to the server 104, such as terminal devices, network platforms, etc. operated by creators, owners, operators, etc. of the digital content.

[0027] The server 104 and the terminal device 112 can collaborate to reconstruct a corresponding virtual space based on the two-dimensional content provided by the two-dimensional content source 102, and embed digital content selected from the digital content provided by the digital content provider 110 in the virtual space. The server 104 can be a remote computer or computer system relative to the user 122. The terminal device 112 can be any local device that can provide a virtual space experience to the user, which can also be referred to as a virtual space application device. Preferably, in order to provide an immersive experience to the user, the terminal device 112 can be a head-mounted display device that can guide the user to have a multi-sensory experience of vision, hearing, touch, etc. in the virtual space. Taking a head-mounted display device that uses VR technology to provide a virtual space as an example, it can also be referred to as VR glasses, VR goggles, VR helmets, etc. in terms of its appearance. The server 104 may include, for example, a processing unit 106, a storage unit 108, etc. The terminal device 112 may include, for example, a presentation unit 114, a behavior recognition unit 116, a processing unit 118, a storage unit 120, etc.

[0028] The storage unit 108 in the server 104 may, for example, store the two-dimensional content obtained from the two-dimensional content source 102. The processing unit 106 in the server 104 may, for example, implement processes related to virtual space reconstruction, digital content selection, etc. For example, the processing unit 106 may use, for example, a three-dimensional reconstruction technology to reconstruct the two-dimensional content into an initial virtual space, and provide the initial virtual space to the terminal device 112. The processing unit 118 in the terminal device 112 may further reconstruct the initial virtual space into a virtual space corresponding to the user 122 based on data collected on site, such as information of the user 122. For example, the initial virtual space may be transformed into a virtual space suitable for the user 122 based on the current position, viewing angle, etc. of the user 122.

[0029] The storage unit 108 in the server 104 may also store digital content obtained from the digital content provider 110. The processing unit 106 may preliminarily select a predetermined number of candidate digital content from the digital content stored in the storage unit 108. The preliminary selection may be performed for a specific user and based on the basic information of the user. The basic information of the user may include, for example, one or more of the user profile, operation history, social network information, and any other user-related information of the user. In this article, the user profile may refer to various information about the user, such as identifier, gender, age, preferences, etc. The operation history may include information associated with the user's past operations, such as clicked digital content, purchased goods, etc. Social network information may include information related to the user's friends, colleagues, relatives and friends. Taking the preliminary selection of digital content for the user 122 of the terminal device 112 as an example, the processing unit 106 may sort the digital content provided by the digital content provider 110 based on the basic information of the user 122, and select a predetermined number of top-ranked digital content from them. The group of digital content may be provided to the terminal device 112 as a group of candidate digital content for embedding in the virtual space.

[0030] The terminal device 112 may receive the set of candidate digital content from the server 104 in advance, for example, periodically, before presenting the virtual space, when the user logs in, etc. In this way, the usage of bandwidth and / or computing resources when presenting the virtual space may be reduced, which is particularly beneficial in the case of a live-based virtual space. The terminal device 112 may store the set of candidate digital content in the storage unit 120 for subsequent use.

[0031] The processing unit 118 in the terminal device 112 may embed digital content in the virtual space, for example, may determine the digital content to be added to the virtual space from the set of candidate digital content and add the determined digital content to the virtual space. For example, the digital content to be added to the virtual space may be determined based at least on information associated with the user's behavior in the virtual space. Figure 2 To illustrate an exemplary process of embedding digital content in a virtual space.

[0032] The behavior of the user in the virtual space may include, for example, the behavior related to the existing digital content performed by the user in the virtual space, which may also be referred to as digital content interaction behavior. The digital content interaction behavior may include, for example, watching, walking towards, touching the digital content, etc. Such behavior may indicate whether the user is currently interested in the existing digital content. The behavior of the user in the virtual space may also include, for example, the behavior related to the environmental objects performed by the user in the virtual space, which may also be referred to as environmental interaction behavior. Environmental interaction behavior may include, for example, walking towards the stage, waving to the singer, etc. Such behavior may indicate the current characteristics of the user, such as personality, hobbies, habits, etc.

[0033] Behavior information associated with the behavior can be extracted. Behavior information can include, for example, various types of information associated with the behavior, such as type, start time, duration, pause time, starting position, direction, amplitude, etc. The type can indicate what category the action associated with the behavior belongs to, including, for example, watching, walking, waving, touching, etc.; the start time can indicate the time when the behavior starts to occur; the duration can indicate how long the behavior lasts; the pause time can indicate how long the behavior is paused during the behavior; the start position can indicate the position where the behavior starts; the direction can indicate the direction in which the behavior is proceeding; the amplitude can indicate the degree of progress of the behavior; and so on. The behavior "walking to the stage" is used as an example to illustrate these behavior information. The type of the behavior can be "walking", the start time can be when the user starts walking, the duration can be how long the user walks, the pause time can be how long the user pauses during walking, the starting position can be the position where the user starts walking, the direction can be "towards the stage", and the amplitude can be the distance walked; and so on. In addition, for digital content interaction behaviors related to existing digital content, the behavior information associated with it can also include an identifier of the digital content. Information associated with the digital content interaction behavior may be referred to as digital content interaction behavior information, and information associated with the environment interaction behavior may be referred to as environment interaction behavior information.

[0034] The behavior of the user 122 in the virtual space can be detected by the behavior recognition unit 116 in the terminal device 112, and the behavior information associated with these behaviors can be obtained. The behavior recognition unit 116 may include, for example, sensors, cameras, etc. The sensors may include, for example, gravity sensors, position sensors, eye tracking sensors, etc. The gravity sensor can use the characteristic that its internal crystals will deform due to acceleration to detect actions related to acceleration, such as tilting, lying down, going uphill, going downhill, etc. The position sensor can detect the user's position and assist in detecting behaviors related to position changes, such as walking, running, etc. The eye tracking sensor can track the user's line of sight to detect the user's viewing behavior. The camera can be used to detect the movements of the user's torso and / or limbs, such as raising hands, waving, lifting legs, etc.

[0035] The virtual space embedded with digital content can be presented to the user 122 through the presentation unit 114 in the terminal device 112. The presentation unit 114 can be a display screen, a monitor, etc. In addition, the presentation unit 114 can also present other information to the user 122, such as a login interface, usage prompts, etc.

[0036] It should be understood that all network entities included in the architecture 100 are exemplary. According to actual application scenarios and requirements, the architecture 100 may include more or fewer network entities, and these network entities may be combined and divided in any manner. For example, multiple units in the terminal device 112 may be separated in position, for example, the behavior recognition unit 116 may be separated in position from other units in the terminal device 112. In addition, although only one terminal device 112 is shown in the architecture 100, there may be different numbers of terminal devices connected to the server 104 through the network. In addition, although the digital content provider 110 is shown as a single network entity, it may also represent multiple network entities that can provide digital content.

[0037] Figure 2 An exemplary process 200 for embedding digital content in a virtual space according to an embodiment of the present disclosure is shown. The process 200 may be executed by a terminal device, for example, Figure 1 206 and / or the behavior information 224 associated with the user's behavior in the virtual space 232. Through the process 200, digital content 230 can be selected from a set of candidate content 202 based on the basic features 214 corresponding to the basic information 206 of the user and / or the behavior information 224 associated with the user's behavior in the virtual space 232, and the selected digital content 230 can be added to the virtual space 232 to obtain an updated virtual space 236. When selecting digital content, each candidate digital content in the set of candidate content 202 can be scored by the scoring model 228, and the candidate digital content with the highest score can be selected as the digital content 230.

[0038] A set of candidate digital content 202 and a set of candidate digital content features 204 corresponding thereto may be obtained. A set of candidate digital content 202 may include n candidate digital content, such as candidate digital content 202-1, candidate digital content 202-2, ..., candidate digital content 202-n. A set of candidate digital content features 204 may include candidate digital content features corresponding to each candidate digital content in a set of candidate digital content 202, such as candidate digital content features 204-1, candidate digital content features 204-2, ..., candidate digital content features 204-n. In this article, a feature may refer to a set of information generated based on raw data in a form that is conducive to deep learning model processing. Taking candidate digital content feature 204-i (1≤i≤n) as an example, it may be a set of information generated based on candidate digital content 202-i in a form that is conducive to scoring model 228 processing. Preferably, each candidate digital content feature may be generated at a server and provided to a terminal device together with the corresponding candidate digital content.

[0039] The basic features 214 may include, for example, user profile features 216, operation history features 218, social network features 220, etc., each of which may be generated based on corresponding information in the user's basic information 206. For example, the user profile features 216 may be generated based on the user profile 208, the operation history features 218 may be generated based on the operation history 210, the social network features 220 may be generated based on the social network information 212, etc. Each basic feature may be generated at the terminal device. Alternatively, each basic feature may also be generated at the server and provided to the terminal device.

[0040] At 222, a behavior recognition operation may be performed, for example, detecting the user's behavior in the virtual space 232 and obtaining behavior information 224 associated with the behavior. For example, the behavior may be detected by a behavior recognition unit in the terminal device and information associated with the behavior may be obtained.

[0041] At 226, digital content 230 to be added to the virtual space 232 may be selected from the set of candidate digital content 202 based on the behavior information 224 and / or the basic features 214. In one embodiment, each candidate digital content in the set of candidate digital content 202 may be scored by a scoring model 228, and the candidate digital content with the highest score in the set of candidate digital content 202 may be selected as the digital content 230. Figure 3 To illustrate the exemplary structure of the scoring model 228 and the corresponding scoring process.

[0042] After selecting the digital content 230, at 234, the digital content 230 can be added to the virtual space 232 to obtain an updated virtual space 236. In one embodiment, the digital content can be added to a predefined position in the virtual space. For example, a set of positions suitable for adding digital content can be pre-detected and / or created. Taking the reconstruction of a virtual space based on a video and adding two-dimensional digital content in the virtual space as an example, some quadrilateral areas, such as billboards, picture frames, display screens, etc., can be pre-detected from the video. Alternatively or additionally, a set of quadrilateral areas can be pre-created in the video, for example, quadrilateral areas can be created in an open area detected in the video, wherein the open area can refer to an area that does not contain significant environmental objects or in which the environmental objects do not produce obvious relative motion, such as a wall, a ground, a table, a stadium stand, etc. The pre-detected or pre-created quadrilateral area can be used as a position for adding two-dimensional digital content. In addition, the open area detected in the video can also be used as a pre-defined position suitable for adding three-dimensional digital content. Pre-defining locations suitable for adding digital content can reduce the usage of bandwidth and / or computing resources when presenting the virtual space. In another embodiment, locations suitable for adding virtual content in the virtual space can be determined in a real-time and dynamic manner. For example, for a virtual space based on a live video, the content appearing in the live video is not completely known before the live video begins. In this case, a quadrilateral area suitable for adding two-dimensional digital content can be detected or created from the video in real-time and dynamically, and / or an open area suitable for adding three-dimensional digital content can be detected.

[0043] It should be understood that Figure 2 The process 200 in the figure is merely an example of a process for embedding digital content. Depending on the actual application requirements, the process for implementing the embedding of digital content may include any other steps, and may include more or fewer steps. For example, when selecting digital content, it is also feasible to only consider the user's behavior information without considering its basic characteristics. In addition, the specific order or hierarchy of the steps in the process 200 is only exemplary, and the process for embedding digital content may be performed in an order different from the described order.

[0044] Figure 3 An exemplary scoring model 300 according to an embodiment of the present disclosure is shown. The scoring model 300 may be located in a terminal device, for example, Figure 1 in the terminal device 112, and may correspond to, for example Figure 2The scoring model 228 in the scoring model 300. The scoring model 300 can score each candidate digital content in a set of candidate digital content based on the user's behavior information 302 and / or basic features 308. For example, the user's behavior information 302, the basic features 308 corresponding to the user's basic information, the specific candidate digital content features corresponding to the specific candidate digital content, etc. can be provided as input to the scoring model 300. The processing process of the scoring model 300 is described below by taking the candidate digital content features 316 corresponding to a certain candidate digital content as an example. The scoring model 300 can process the behavior information 302, the basic features 308, the candidate digital content features 316, etc. to generate a score 344 corresponding to the candidate digital content corresponding to the candidate digital content features 316.

[0045] The behavior information 302 may include, for example, digital content interaction behavior information 304, environment interaction behavior information 306, etc. The basic features 308 may include, for example, user profile features 310, operation history features 312, social network features 314, etc.

[0046] The digital content interaction behavior information 304 may be an information set associated with one or more digital content interaction behaviors of the user in the virtual space. Each information subset in the information set corresponds to a digital content interaction behavior, and may include various types of behavior information associated with the digital content interaction behavior, such as a digital content identifier, type, start time, duration, pause time, starting position, direction, amplitude, etc. The digital content interaction behavior features 324 corresponding to the digital content interaction behavior information 304 may be generated by the embedding layer 318, the fully connected layer 320, and the recurrent neural network (RNN) layer 322 in the scoring model 300. Similarly, the environment interaction behavior information 306 may be an information set associated with one or more environment interaction behaviors of the user in the virtual space. Each information subset in the information set corresponds to an environment interaction behavior, and may include various types of behavior information associated with the environment interaction behavior, such as type, start time, duration, pause time, starting position, direction, amplitude, etc. The environment interaction behavior feature 332 corresponding to the environment interaction behavior information 306 can be generated by the embedding layer 326, the fully connected layer 328 and the RNN layer 330 in the scoring model 300. Preferably, the RNN layer 322 / 330 can be an RNN layer with an attention mechanism introduced to improve the performance of feature generation for long sequence inputs.

[0047] The digital content interaction behavior features 324 , the environment interaction behavior features 332 , the basic features 308 , and the candidate digital content features 316 may be concatenated through a concatenation and flattening layer 334 , and the concatenated features may be dimensionally reduced through a flattening operation to obtain a joint representation 336 .

[0048] At the multi-task layer 338, the joint representation 336 can be scored by constructing multiple tasks corresponding to multiple indicators to obtain multiple sub-scores 340 for the candidate digital content corresponding to the candidate digital content feature 316. In this article, the scores corresponding to each task can be referred to as sub-scores. The indicator can include, for example, the type and / or duration of the behavior of the user interacting with the candidate digital content. The various tasks in the multi-task layer 338 can respectively predict the values ​​of these indicators of the candidate digital content and score the candidate digital content based on the predicted values. As an example, for the indicator of the type of behavior of the user interacting with the candidate digital content, the corresponding task can be to predict the type of behavior of the user interacting with the candidate digital content corresponding to the candidate digital content feature 316, and generate a sub-score corresponding to the predicted type. For example, the behavior type "watching" can have a higher sub-score, while the behavior type "waving" may have a lower sub-score. As another example, for an indicator regarding the duration of a user's behavior of interacting with candidate digital content, a corresponding task may be to predict the duration of a user's behavior of interacting with candidate digital content corresponding to candidate digital content feature 316, and generate a score corresponding to the predicted duration. For example, a sub-score corresponding to a longer duration may be higher, while a sub-score corresponding to a shorter duration may be lower.

[0049] At the aggregation layer 342, the plurality of sub-scores 340 outputted by the multi-task layer 338 may be aggregated to obtain a score 344 for the candidate digital content corresponding to the candidate digital content feature 316. In one embodiment, the score 344 may be obtained by directly summing the plurality of sub-scores 340. In another embodiment, the score 344 may be obtained by weighted summing the plurality of sub-scores 340. For example, different weights may be set for each indicator according to actual application requirements, for example, a larger weight may be set for an indicator of greater concern, so that the candidate digital content having a better performance of the indicator may be selected.

[0050] It should be understood that Figure 3 The scoring model 300 shown is only an example of a scoring model. According to actual application requirements, the scoring model may have any other structure and may include more or fewer layers.

[0051] The embodiments of the present disclosure propose a scoring model for scoring candidate digital content, such as Figure 3 The scoring model 300 in is obtained by training an initial scoring model at a group of terminal devices of a group of users. Figure 4 An exemplary process 400 for training a scoring model according to an embodiment of the present disclosure is shown. In process 400, an initial scoring model may be distributed to a group of terminal devices of a group of users. Each terminal device in the group of terminal devices may be any device suitable for providing a virtual space experience, such as VR glasses, VR helmets, etc. A user may use his terminal device to enter a virtual space embedded with a set of digital content. Each user's behavior for each digital content may be used to train a corresponding initial scoring model. The scoring models trained at each terminal device may be aggregated into an updated scoring model.

[0052] First, the initial scoring model 402 can be distributed to a group of terminal devices 404 of a group of users, including, for example, terminal device 404-1, terminal device 404-2, ..., terminal device 404-n. The behavior of each user can be used to train the initial scoring model in its terminal device. The following takes any terminal device, such as terminal device 404-i, as an example to illustrate an exemplary process of training a scoring model.

[0053] The user of the terminal device 404-i can use the terminal device to enter the virtual space and interact with it. A set of digital content can be embedded in the virtual space. The user's behavior on each digital content can be detected and the corresponding behavior information can be obtained. Subsequently, training samples can be generated based on the behavior information. A set of training samples corresponding to a set of digital content can be combined into a training set, and the training set can be used to train the initial scoring model in the terminal device 404-i to obtain a trained scoring model 406-i.

[0054] After obtaining a group of trained scoring models 406, such as trained scoring model 406-1, trained scoring model 406-2, ..., trained scoring model 406-n, the group of trained scoring models 406 may be aggregated into an updated scoring model 408. Preferably, a fully trained scoring model may be selected from the group of trained scoring models 406 for aggregation. For example, a trained scoring model may be selected that is highly used by users and thus trained with more training samples.

[0055] The updated scoring model 408 can be directly deployed for scoring candidate digital content. Preferably, process 400 can be performed iteratively. For example, after obtaining the updated scoring model 408, it can be distributed to a group of terminal devices 404, and the above process is performed again until the performance of the model converges.

[0056] In process 400, user behavior is only used for training locally by the user and will not be uploaded to the server, so leakage of user privacy can be avoided. In addition, the updated scoring model obtained by aggregating the models of each user does not have information strongly related to a specific user, so user privacy can be further protected.

[0057] It should be understood that Figure 4 The process 400 in the figure is merely an example of a process for scoring a model. Depending on the actual application requirements, the process for scoring a model may include any other steps, and may include more or fewer steps. In addition, the specific order or hierarchy of the steps in the process 400 is merely exemplary, and the process for scoring a model may be performed in an order different from the order described.

[0058] Figure 5 An exemplary terminal device 500 according to an embodiment of the present disclosure is shown. The terminal device 500 may have, for example, the appearance of glasses. The terminal device 500 can present a virtual space and other information such as a login interface, usage prompts, etc. to the user through a presentation unit (not shown). The presentation unit may be a display screen, a display, etc., which may be placed in an area facing the user's eyes in the terminal device 500. In addition, the user's behavior in the virtual space may be detected by a behavior recognition unit and behavior information associated with these behaviors may be obtained. The behavior recognition unit may include, for example, a gravity sensor 502, a position sensor 504, and cameras 506, 508, 510, and 512, etc.

[0059] The gravity sensor 502 can use the characteristic that its internal crystal will deform due to acceleration to detect actions related to acceleration, such as tilting, lying down, going uphill, going downhill, etc. The position sensor 504 can detect the position of the user and assist in detecting behaviors related to position changes, such as walking, running, etc. The camera can be used to detect the actions of the user's trunk and / or limbs. For example, the cameras 506 and 508 can be placed in the front of the terminal device 500, and their orientation can be suitable for detecting actions occurring in front of the user, such as hand movements such as raising hands and waving hands. In addition, the cameras 506 and 508 can also detect environmental objects, digital content, etc. in the virtual space. Preferably, the cameras 506 and 508 can be placed on the left and right sides of the front of the terminal device 500, respectively, so as to detect the actions of the left hand and the right hand, respectively. The cameras 510 and 512 can be placed at the bottom of the terminal device 500, and their orientation can be suitable for detecting actions occurring in front of or below the user, such as the actions of the user's trunk and / or limbs such as walking, kicking, turning, etc.

[0060] These behavior recognition units can determine the user's behavior based on the detected actions. In one embodiment, the behavior can be jointly determined based on the actions detected by the various behavior recognition units. For example, when the camera 510 and / or 512 detects that the user is walking, and the position sensor 504 detects that the user's position is changing and approaching the stage, it can be determined that the user's behavior is "walking towards the stage." After the user's behavior is determined, the behavior information associated with the behavior can be extracted accordingly.

[0061] It should be understood that Figure 5 The terminal device 500 in the figure is only an example of a terminal device. According to actual application requirements, the terminal device may include any other units and may include more or fewer units. For example, the terminal device may also include a processing unit for determining digital content based at least on the behavior information and adding the digital content in the virtual space, a storage unit for storing a set of candidate digital content, and the like. In addition, Figure 5 The positions and numbers of the various behavior recognition units shown in are exemplary. Depending on the actual application requirements, the various behavior recognition units may be located at other locations and may have other numbers. In addition, the terminal device 500 may integrate the behavior recognition unit with other units, so that the user can switch between the real environment and the virtual space using only a single device, thereby obtaining a more realistic immersive experience. It should be understood that one or more of these behavior recognition units may also be separated in position from other units in the terminal device 500, for example, some sensors or cameras may also be located outside the terminal device 500.

[0062] Figure 6is a flow chart of an exemplary method 600 for embedding digital content in a virtual space according to an embodiment of the present disclosure.

[0063] At 610, user behavior in the virtual space can be detected.

[0064] At 620, behavior information associated with the behavior can be obtained.

[0065] At 630, digital content can be determined based at least on the behavioral information.

[0066] At 640, the digital content can be added in the virtual space.

[0067] In one implementation, the virtual space may be constructed using VR technology.

[0068] In one implementation, the behavior may include at least one of the following: a digital content interaction behavior related to existing digital content performed by the user in the virtual space; and an environmental interaction behavior related to environmental objects performed by the user in the virtual space.

[0069] In one embodiment, the behavior information may include at least one of the following information associated with the behavior: type, start time, duration, pause time, start position, direction, and amplitude. In the case where the behavior includes a digital content interaction behavior related to existing digital content performed by the user in the virtual space, the behavior information may also include an identifier of the existing digital content.

[0070] In one embodiment, the determining of the digital content may also be based on at least one of a user profile, an operation history, and social network information of the user.

[0071] In one implementation, the virtual space may be presented by a terminal device, and the behavior may be detected by a behavior recognition unit in the terminal device.

[0072] The behavior recognition unit may include a gravity sensor and / or a position sensor.

[0073] The behavior recognition unit may include at least one camera, which is oriented to detect movements of the user's torso and / or limbs.

[0074] In one embodiment, the determining the digital content may include: selecting the digital content from a group of candidate digital content based at least on the behavior information.

[0075] The selecting the digital content may include: for each candidate digital content in the set of candidate digital contents, scoring the candidate digital content based at least on the behavior information; and selecting the candidate digital content with the highest score in the set of candidate digital contents as the digital content.

[0076] Scoring the candidate digital content may include: predicting the type and / or duration of the user's behavior of interacting with the candidate digital content based at least on the behavior information and the candidate digital content; and calculating the score of the candidate digital content based on the predicted type and / or duration.

[0077] The scoring of the candidate digital content may be performed by a deep learning model in a terminal device. The deep learning model may be obtained by training an initial deep learning model at a group of terminal devices of a group of users.

[0078] The set of candidate digital contents may be received from the server in advance and stored in the terminal device.

[0079] In one implementation, adding the digital content may include: adding the digital content at a predefined location in the virtual space.

[0080] In one implementation, method 600 may be executed at a terminal device.

[0081] It should be understood that method 600 may also include any steps / processes for embedding digital content in a virtual space according to the above-mentioned embodiments of the present disclosure.

[0082] Figure 7 An exemplary virtual space application device 700 according to an embodiment of the present disclosure is shown.

[0083] The virtual space application device 700 may include: a presentation unit 710, used to present a virtual space; a behavior recognition unit 720, used to detect the user's behavior in the virtual space and obtain behavior information associated with the behavior; and a processing unit 730, used to determine digital content based at least on the behavior information and add the digital content in the virtual space.

[0084] In one embodiment, the behavior recognition unit 720 may include at least one of the following: a gravity sensor; a position sensor; and at least one camera, which is oriented to detect the movements of the user's torso and / or limbs.

[0085] In one implementation, the virtual space application device 700 may further include: a storage unit 740, configured to store a group of candidate digital contents. The processing unit 730 may further be configured to determine the digital content from the group of candidate digital contents based at least on the behavior information.

[0086] It should be understood that the virtual space application device 700 may also include any other units configured to embed digital content in a virtual space according to the above-mentioned embodiments of the present disclosure. In addition, the various units included in the virtual space application device 700 may also perform any relevant operations for embedding digital content in a virtual space according to the above-mentioned embodiments of the present disclosure.

[0087] Figure 8 An exemplary apparatus 800 for embedding digital content in a virtual space according to an embodiment of the present disclosure is shown.

[0088] The apparatus 800 may include at least one processor 810. The apparatus 800 may also include a memory 820 connected to the processor 810. The memory 820 may store computer executable instructions, which, when executed, cause the processor 810 to perform the following operations: receive behavior information associated with a user's behavior in the virtual space; determine digital content based at least on the behavior information; and add the digital content in the virtual space.

[0089] It should be understood that the processor 810 may also perform any operation for embedding digital content in a virtual space according to the above-mentioned embodiments of the present disclosure.

[0090] An embodiment of the present disclosure proposes a computer program product for embedding digital content in a virtual space, including a computer program, which is executed by at least one processor to: receive behavior information associated with a user's behavior in the virtual space; determine digital content based at least on the behavior information; and add the digital content in the virtual space.

[0091] It should be understood that the computer program can also be executed to implement any operation for embedding digital content in a virtual space according to the above-mentioned embodiments of the present disclosure.

[0092] The embodiment of the present disclosure proposes a device for embedding digital content in a virtual space. The device may include: a behavior information receiving module for receiving behavior information associated with a user's behavior in the virtual space; a digital content determining module for determining digital content based at least on the behavior information; and a digital content adding module for adding the digital content in the virtual space.

[0093] It should be understood that the device may also include any other modules configured to embed digital content in a virtual space according to the above-mentioned embodiments of the present disclosure.

[0094] Embodiments of the present disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform any operation of the method for embedding digital content in a virtual space according to the embodiments of the present disclosure as described above.

[0095] It should be appreciated that all operations in the method described above are merely exemplary, and the present disclosure is not limited to any operations in the method or the order of these operations, but should cover all other equivalent changes under the same or similar concept.

[0096] It should also be appreciated that all modules in the above described device can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. In addition, any module in these modules can be further divided into submodules or combined together in function.

[0097] Processors have been described in conjunction with various devices and methods. These processors can be implemented using electronic hardware, computer software or any combination thereof. Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraints imposed on the system. As an example, the processor provided in the present disclosure, any part of the processor or any combination of processors can be realized using a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gated logic unit, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The function of the processor provided in the present disclosure, any part of the processor or any combination of processors can be realized using software executed by a microprocessor, a microcontroller, a DSP or other suitable platforms.

[0098] Software should be broadly considered to mean instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software can reside in a computer-readable medium. A computer-readable medium can include, for example, a memory, which can be, for example, a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic stripe), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although the memory is shown as being separated from the processor in the multiple aspects provided in the present disclosure, the memory can also be located inside the processor, such as a cache or a register.

[0099] The above description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein. All structural and functional equivalents to the elements of the various aspects described in this disclosure that are known or to be known to those of ordinary skill in the art are expressly incorporated herein and covered by the claims.

Claims

1. A method for embedding digital content in a virtual space, comprising: detecting the user's behavior in the virtual space; Obtaining behavior information associated with the behavior; Determining digital content based at least on the behavior information, wherein determining the digital content comprises: For each candidate digital content in a set of candidate digital contents, scoring the candidate digital content by: predicting the type and duration of the user's behavior of interacting with the candidate digital content based on at least the behavior information and the candidate digital content, and calculating the score of the candidate digital content based on the predicted type and duration; and Selecting the candidate digital content with the highest score among the set of candidate digital content as the digital content; and The digital content is added in the virtual space.

2. The method according to claim 1, wherein: The virtual space is constructed using virtual reality (VR) technology.

3. The method according to claim 1, wherein: The behavior includes at least one of the following: Digital content interaction behavior related to existing digital content performed by the user in the virtual space; and The user performs environmental interaction behaviors related to environmental objects in the virtual space.

4. The method according to claim 1, wherein: The behavior information includes at least one of the following information associated with the behavior: type, start time, duration, pause time, start position, direction, and amplitude.

5. The method according to claim 4, wherein: The behavior includes a digital content interaction behavior related to the existing digital content performed by the user in the virtual space, and the behavior information also includes an identifier of the existing digital content.

6. The method according to claim 1, wherein: The determining of the digital content is also based on at least one of a user profile, an operation history, and social network information of the user.

7. The method according to claim 1, wherein: The virtual space is presented by a terminal device, and the behavior is detected by a behavior recognition unit in the terminal device.

8. The method according to claim 7, wherein: The behavior recognition unit includes a gravity sensor and / or a position sensor.

9. The method according to claim 7, wherein: The behavior recognition unit includes at least one camera, which is oriented to detect movements of the user's torso and / or limbs.

10. The method according to claim 1, wherein: The scoring of the candidate digital content is performed by a deep learning model in a terminal device, and the deep learning model is obtained by training an initial deep learning model at a group of terminal devices of a group of users.

11. The method according to claim 1, wherein: The set of candidate digital contents is received in advance from the server and stored in the terminal device.

12. The method according to claim 1, wherein: Adding the digital content comprises: The digital content is added at a predefined location in the virtual space.

13. The method according to claim 1, wherein: The method is executed at a terminal device.

14. A virtual space application device, comprising: A presentation unit, used for presenting a virtual space; a behavior recognition unit, configured to detect the behavior of the user in the virtual space and obtain behavior information associated with the behavior; as well as A processing unit is configured to determine digital content based at least on the behavior information, and add the digital content to the virtual space, wherein the determining the digital content comprises: For each candidate digital content in a set of candidate digital contents, scoring the candidate digital content by: predicting the type and duration of the user's behavior of interacting with the candidate digital content based on at least the behavior information and the candidate digital content, and calculating the score of the candidate digital content based on the predicted type and duration; and The candidate digital content with the highest score among the set of candidate digital content is selected as the digital content.

15. The device according to claim 14, wherein: The behavior recognition unit includes at least one of the following: Gravity sensor; Position sensor; as well as At least one camera is oriented to detect movement of the torso and / or limbs of the user.

16. A computer program product for embedding digital content in a virtual space, comprising a computer program, the computer program being executed by at least one processor for: receiving behavior information associated with a user's behavior in the virtual space; The digital content is determined based at least on the behavior information, wherein: Determining the digital content includes: For each candidate digital content in a set of candidate digital contents, scoring the candidate digital content by: predicting the type and duration of the user's behavior of interacting with the candidate digital content based on at least the behavior information and the candidate digital content, and calculating the score of the candidate digital content based on the predicted type and duration; and Selecting the candidate digital content with the highest score among the set of candidate digital content as the digital content; and The digital content is added in the virtual space.

Images