Virtual space generator

Abstract

To provide a virtual space generating device having unprecedented gameplay. [Solution] A virtual space generation device (10) comprising a display data memory unit (112, 113) in which display data for the virtual space and display data for the player character are stored, a virtual space generation unit (22) that generates a virtual space using the display data for the virtual space, a player character action processing unit (23) that uses the display data for the player character to move the player character in the virtual space in response to external input operations, an image display processing unit (27) that generates an image in the virtual space based on data of the image including a target area input from the outside, a judgment unit (28) that judges whether a predetermined action by the player character has been performed against the target area, and a judgment result output unit (29) that outputs the result of the judgment by the judgment unit.

Description

[Technical field]

[0001] The present invention relates to a virtual space generating device. [Background technology]

[0002] In recent years, technology has been widely used that allows users to control player characters in a virtual space to participate in events, games, etc. By using virtual reality (VR) technology, in which a user wears a device such as a head-mounted display (HMD) and the display shows an image of the virtual space from the player character's point of view, the user can enjoy events, games, etc. while feeling immersed in the experience.

[0003] Patent Document 1 describes a system in which a screen object is placed on a stage set up in a virtual space, and images of live performances, sports, plays, etc., filmed in the real space are displayed on the screen object. In this system, a user operates a player character that operates in the virtual space, and an image of the virtual space captured from the player character's viewpoint is displayed on a head-mounted display worn by the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-68643 [Non-patent literature]

[0005] [Non-Patent Document 1] "[Generative AI] A thorough explanation of the features and use cases of 44 major free and paid tools for LLM, image generation, code generation, video generation, etc.!", [online], September 12, 2024, AI Market Editorial Department, [Searched September 27, 2024], Internet<URL:https: / / ai-market.jp / services / generative-ai-tools / > [Non-Patent Document 2] "What is YOLO? Explaining the differences with other methods and the advantages and disadvantages", [online], February 7, 2024, AIsmiley Editorial Department, [Searched September 27, 2024], Internet<URL:https: / / aismiley.co.jp / ai_news / yolo / > [Non-Patent Document 3] "An easy-to-understand explanation of object detection! A detailed introduction to the differences, scope of use, and methods", [online], Future Standard Co., Ltd., [Searched September 27, 2024], Internet<URL:https: / / www.scorer.jp / blog / object-detection> [Non-Patent Document 4] Miyazawa Riyuya,"StrongSORT: DeepSORT is back and stronger! Upgraded tracking model!",[online],December 31, 2022,wevnal Inc.,[Retrieved September 27, 2024],Internet<URL:https: / / ai-scholar.tech / articles / object-tracking / strongsort> [Non-Patent Document 5] "Hakuhodo DY Media Partners' AaaS Tech Lab develops technology that uses generative AI to detect any object in an image with high accuracy using natural language ~ To be used in RoomClip's services and the media business field ~", [online], May 14, 2024, Hakuhodo DY Media Partners Inc., [Searched September 27, 2024], Internet<URL:https: / / www.hakuhodody-media.co.jp / newsrelease / service / 20240514_34939.html> Summary of the Invention [Problem to be solved by the invention]

[0006] In the system described in Patent Document 1, the user merely passively enjoys the images displayed on the screen objects, and no changes are produced by moving the player character, so the system lacks gameplay.

[0007] The problem to be solved by the present invention is to provide a virtual space generating device having gameplay not previously seen. [Means for solving the problem]

[0008] The virtual space generating device according to the present invention, which has been made to solve the above problems, is: a display data storage unit in which display data of a virtual space and display data of a player character operating in the virtual space are stored; a virtual space generation unit that generates a virtual space using the display data of the virtual space stored in the display data storage unit; a player character operation processing unit that uses the display data of the player character stored in the display data storage unit to cause the player character to operate in the virtual space in response to an external input operation; an image display processing unit that generates an image including a target area in the virtual space based on data of the image; a determination unit that, when a predetermined action is performed by the player character on the image, determines whether the action is performed on the target area; a judgment result output unit that outputs a judgment result by the judgment unit; Equipped with.

[0009] In the virtual space generating device according to the present invention, an image display processing unit generates an image including a target area in a virtual space. When a user performs an input operation to the virtual space generating device using an external input device to move a player character, a player character movement processing unit moves the player character in the virtual space in response to the input operation. The external input device is, for example, a head mounted display (HMD), and an image of the virtual space captured from the viewpoint of the player character is displayed on the display unit.

[0010] Data of an image to be generated in a virtual space is input from the outside. Information for identifying the position of the target area in the image may or may not be included in the data of the image input from the outside (i.e., the target area included in the image may be undefined at the time when the image display processing unit generates the image). In the latter case, the target area may be identified using, as the determination unit, a unit including a classifier using a trained model in which learning data consisting of a pair of game images and target areas is trained by machine learning in advance for various types of games. In addition, this image data may be prepared in advance, or may be generated by a large-scale language model (LLM) or the like based on keywords input by the user. In the latter case, the determination unit may also include a classifier using an LLM.

[0011] When a player character operated by a user performs a predetermined action on an image generated in a virtual space, the determination unit determines whether or not the action is performed on a target area, and the determination result output unit outputs the determination result. For example, if the generated image is related to a shooting game, the predetermined action is a shooting action by the player character, and the determination unit determines whether or not a bullet fired by the action of the player character hits a target area (enemy character, airplane, animal, etc.) included in the image. Alternatively, for example, if the image generated in a virtual space is related to a dance game, the predetermined action is a dance action by the player character, and the determination unit determines the degree of agreement between the body movement of the player character and the target area (the dancer's body) included in the image. This allows the user to enjoy an unprecedented game in which the player character moves according to the image generated in a virtual space. Effect of the Invention

[0012] By using the virtual space generating device according to the present invention, it is possible to provide a game-like element to the virtual space generating device which places screen objects in a virtual space and displays images thereon. [Brief description of the drawings]

[0013] [Figure 1] 1 is a diagram showing the configuration of a main part of a game system including an embodiment of a virtual space generating device according to the present invention. [Diagram 2] 11A and 11B are diagrams illustrating a state in which an action is performed by a player character on an image displayed on a screen object in a shooting game using the virtual space generating device of this embodiment. [Diagram 3] 13 shows an example of generating a virtual space based on an image of a screen object in the virtual space generating device of the present embodiment. [Figure 4] 11A to 11C are diagrams illustrating judgments using a judgment panel in a shooting game using the virtual space generating device of the present embodiment. [Diagram 5] 13A to 13C are diagrams illustrating a state in which an action is performed by a player character on an image displayed on a screen object in a dance game using the virtual space generating device of this embodiment. [Figure 6] 11A to 11C are diagrams illustrating judgment using a judgment panel in a dance game using the virtual space generating device of this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] An embodiment of a virtual space generating device according to the present invention will be described below with reference to the drawings.

[0015] Fig. 1 is a diagram showing the main configuration of a game system 1 including a virtual space generating device 10 of this embodiment. The game system 1 includes the virtual space generating device 10 and a user terminal 40. Although only one user terminal 40 is shown in Fig. 1, there may be multiple user terminals 40. Furthermore, the game system 1 and the user terminal 40 are connected to each other via a wireless communication network such as the Internet, or a wired communication network.

[0016] The virtual space generating device 10 includes a storage unit 11. The storage unit 11 includes a user information storage unit 111, a virtual space display data storage unit 112, a character data storage unit 113, an effect data storage unit 114, an image generator storage unit 115, an image classifier storage unit 116, and an image data storage unit 117. Note that the concept of an image in this specification includes not only a still image but also a video (for example, a video composed of still images at 60 frames per second).

[0017] The user information storage unit 111 stores information such as a user ID and password for identifying a user who uses the user terminal 40, the type of player character operated by the user, and a terminal ID for identifying the user terminal 40. The virtual space display data storage unit 112 stores display data for the virtual space. The character data storage unit 113 stores display data for player characters and non-player characters that operate in the virtual space. The effect data storage unit 114 stores data for various effects (audio, image, etc.) to be generated in the virtual space.

[0018] The image generator storage unit 115 stores an image generator that generates an image based on a word or sentence input from the outside. For the image generator, a generative AI such as a large-scale language model (LLM: Large Language Models, for example, Non-Patent Document 1) can be suitably used. The image classifier storage unit 116 stores an image classifier that distinguishes between a target area and a non-target area included in an image input from the outside or an image generated by the image generator, and distinguishes a background included in an image based on a predetermined criterion. For the image classifier, for example, a classifier based on an object detection algorithm such as YOLO (You Only Look Once, for example, Non-Patent Documents 2 and 3) or SSD (Single Shot MultiBox Detector, for example, Non-Patent Document 3), or a trajectory estimation algorithm such as DeepSORT or StrongSORT (for example, Non-Patent Document 4) that estimates the trajectory when an object identified by the classifier is a moving object can be used. In addition, when the image generator is a generative AI such as an LLM, a generative AI such as an LLM (for example, Non-Patent Document 5) can also be used for the image classifier. In addition, an algorithm can be used that identifies moving objects (objects whose position changes between successive images in time) contained in a video as target regions.

[0019] The image data storage unit 117 stores image data that can be used when a user plays a game. The image data stored in the image data storage unit 117 may include both images associated with information specifying the position of a target area included in the image and images not associated with the information. The target area means an area targeted when a user plays a game, and is, for example, a display area of ​​a target object to be shot in a shooting game, and a display area of ​​a model dancer in a dance game. The image data stored in the image data storage unit 117 may include both images associated with data stored in the virtual space display data storage unit 112 and images not associated with the data.

[0020] The virtual space generating device 10 comprises, as functional blocks, a user authentication unit 21, a virtual space generating unit 22, a player character action processing unit 23, an image information input receiving unit 24, an image generating unit 25, an image discrimination unit 26, an image display processing unit 27, a determination unit 28, and an effect generating unit 29. Details of these functional blocks will be described later.

[0021] The virtual space generating device 10 is configured, for example, by a cloud server or a general personal computer, and each of the above-mentioned functional blocks is realized by executing pre-installed dedicated software (virtual space generating program) on a processor.

[0022] The user terminal 40 includes a storage unit 41. The storage unit 41 stores terminal ID information for identifying the user terminal 40 and information about the user who uses the user terminal 40 (user ID information, etc.).

[0023] The user terminal 40 also includes, as hardware, a posture acquisition unit 42, a voice input unit 43, a voice output unit 44, an input unit 45, and a display unit 46. The user terminal 40 is, for example, a head mounted display (HMD), and the posture acquisition unit 42 includes various sensors and a calculation processing unit that determines the position and posture of the user based on the output signals of the sensors. The posture acquisition unit 42, the voice input unit 43, and the voice output unit 44 transmit and receive signals to and from the virtual space generation device 10 at a predetermined frequency (for example, 60 times / second). As a result, information on the user's position and posture acquired by the posture acquisition unit 42 and information on the voice input to the voice input unit 43 are transmitted to the virtual space generation device 10, and are reflected in the position, posture, and voice of the player character in the virtual space. Furthermore, information on the voice generated in the virtual space is output from the voice output unit 44. The display unit 46 is arranged so as to be located in front of the user's eyes when the user wears the user terminal 40 on his / her head. In addition, the user terminal 40 also includes a data processing unit 48 as a functional block.

[0024] Next, the operation of the game system 1 of this embodiment will be described.

[0025] When a user accesses the virtual space generating device 10 by performing a predetermined operation on the user terminal 40, the user authentication unit 21 generates a screen for inputting a user ID and a password, and transmits the data to the user terminal 40. In the user terminal 40, the received data is processed by the data processing unit 48, and the above screen is displayed on the display unit 46. When the user inputs the user ID and password, the user authentication unit 21 identifies the user based on the input user ID and password. Here, the user ID and password are input, but the user ID and password or the terminal ID stored in the storage unit 41 of the user terminal 40 may be automatically transmitted. In the following description, the display data of the screen generated by each unit of the virtual space generating device 10 is transmitted to the user terminal 40, and the data processing unit 48 processes the display data and displays it on the display unit 46. However, the series of processes will not be described repeatedly, and will be simply described as "displaying the screen on the display unit 46", etc.

[0026] When the user authentication unit 21 identifies the user, the virtual space generation unit 22 reads out display data of the virtual space, which is predetermined as an initial state, from the virtual space display data storage unit 112, and generates a virtual space. Furthermore, the player character operation processing unit 23 identifies the type of player character operated by the user based on the data stored in the user information storage unit 111. If multiple types of player characters are associated with the user, the user is prompted to select one of them. Then, the player character is made to appear in the virtual space based on the data stored in the character data storage unit 113. In addition to this, a preset non-player character may also be made to appear.

[0027] Here, the user logs in by inputting a user ID and a password, but the login operation may be omitted. Also, although information on the type of player character operated by the user is stored in advance in user information storage unit 111, information on multiple types of player characters stored in character data storage unit 113 may be presented to the user without being associated with user information, and the user may select a player character to be operated each time. Also, information on the type of player character may be stored in storage unit 41 of user terminal 40, and player character action processing unit 23 may read out the information.

[0028] Next, the image information input receiving unit 24 displays a screen on the display unit 46 that allows the user to select whether to use an image already stored in the image data storage unit 117 or a newly generated image as the image to be used in the game.

[0029] If the user selects to use an existing image, the image information input accepting unit 24 displays a list of images (game footage) stored in the image data storage unit 117 on the display unit 46 and allows the user to select one of them.

[0030] When the user selects one of the images, the image classifier 26 checks whether the data of the selected image is associated with information specifying the position of the target area contained in the image. If the data of the selected image is associated with information specifying the position of the target area, the image classifier is also read. On the other hand, if the data is not associated with information specifying the position of the target area, the image classifier stored in the image classifier storage unit 116 is read and operated.

[0031] Furthermore, the image classifier 26 checks whether the data of the selected image is associated with the display data of the virtual space stored in the virtual space display data storage unit 112. If the data is associated with the data stored in the virtual space display data storage unit 112, the virtual space generation unit 22 reads the data. If the data is not associated with the data stored in the virtual space display data storage unit 112, the image classifier stored in the image classifier storage unit 116 is read and operated. The display data of the virtual space associated with the data of the image is the display data of the virtual space that is displayed in synchronization with the image (game video) displayed on the screen object 5 described below.

[0032] When the user selects to use a newly generated image on the above selection screen, the image generating unit 25 reads out the image generator from the image generator storage unit 115 and asks the user what kind of image (game video) to display. In response, the user inputs words (e.g., shooting game, forest, hunting) or sentences (e.g., a shooting game in which hunting takes place in a forest) by voice or text from the user terminal 40, and the image generator generates image data based on the input voice or text.

[0033] For example, if the image created here is a video of a dance game, when the user plays the dance game, after checking the model movement, the user can generate a dance game in which the user imitates the movement. In that case, the user may input information from the user terminal 40, such as "a dance game consisting of two phases, a model phase (a phase in which the model movement is displayed and the user checks it) and a play phase (a phase in which the user imitates the model movement checked by the user. In this phase, the model movement may or may not be displayed) as a set." Also, in a dance game, if the physique of the model performing the model movement does not match that of the player character, the determination result of the degree of match of the movements may be poor. Therefore, it is advisable to input information on the gender and physique of the player character together with the above information. By inputting such information, the image generating unit 25 can generate an image of the dance game in which a model similar to the physique of the player character performs the model movement. The information input from the user terminal 40 is stored in the storage unit 11 (for example, the image data storage unit 117).

[0034] When the selection or generation of the image to be used in the game is completed, the image display processing unit 27 virtually displays a transparent screen object 5 at a predetermined position in the virtual space 8 (for example, a position directly facing the player character 9 at a distance L from the screen object 5. See FIG. 2.) and displays the image (starts playing the video) based on the selected or generated data. Since the screen object 5 is transparent, when the screen object 5 is captured from the viewpoint of the player character 9, the virtual space behind the screen object 5 can be simultaneously viewed. As described above, the screen object 5 is virtually displayed in the virtual space, and for example, a player character or a non-player character operated by another user can freely pass through (pass through) the screen object 5. In addition, when the image is associated with data stored in the virtual space display data storage unit 112, the virtual space generating unit 22 generates the virtual space based on the display data of the virtual space associated with the image in parallel with the display of the image (playback of the video. The same applies below).

[0035] If the selected or generated image is not associated with the display data of the virtual space stored in the virtual space display data storage unit 112, the image identification unit 26 uses an image identification unit to identify elements contained in the image in parallel with the display of the image, and generates a virtual space 8 that expands the image displayed on the screen object 5. For example, as shown in FIG. 3, if the image has a sky 6 and a ground 7 as a background, the sky 6 includes a sun 61 and clouds 62, and the ground 7 includes a house 71 and a road 72, a virtual space 8 is generated that has the sky 60 in the space above the boundary between the sky 6 and the ground 7, and the ground 70 in the space below the boundary. In addition, clouds 621 are placed in the sky 60, houses 711 are placed on the ground 70, and a road 721 extended from the screen object 5 is displayed, similar to those contained in the image.

[0036] Furthermore, if the data of the selected or generated image does not contain information identifying the position of the target area, the image identification unit 26 identifies the target area 52 contained in the image using an image classifier in parallel with the display of the image (see Figure 2).

[0037] When displaying an image on the screen object 5, the process of identifying the target area 52 may be performed at the same frequency (e.g., 60 times / second) as the frame rate of the image (e.g., 60 frames / second). However, depending on the performance of the devices constituting the virtual space generating device 10 and the type of image classifier, it may be difficult to perform such high-speed processing.

[0038] Therefore, as an image classifier, in addition to object detection algorithms such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector), it is advisable to use trajectory estimation algorithms such as DeepSORT and StrongSORT, which estimate and track the trajectory of an object detected by these object detection algorithms. By adopting such a configuration, it is only necessary to perform object detection at a frequency lower than the frame rate of the video (for example, once per second), and specify the target area 52 based on the position of the object detected every second and the estimated trajectory of the object, making it possible to configure the virtual space generating device 10 with a general personal computer.

[0039] To avoid VR sickness caused by images displayed in a virtual space, a minimum frame rate of 60 frames per second is required, and by using such technology, it is possible to realize games using images with high frame rates. In addition, a wider variety of algorithms can be selected as an image classifier. Alternatively, if all moving objects in the image are targets, a simpler algorithm that detects moving objects and identifies them as the target area 52 may be used.

[0040] When the user operates the player character 9 through the user terminal 40 while an image is displayed (a video is played) on the screen object 5, the player character operation processing unit 23 moves the player character 9 in the virtual space 8 in accordance with the input operation. For example, the player character 9 moves in the virtual space in accordance with the movement of the user, and when the user changes his / her posture, the player character 9 also changes to the same posture.

[0041] When a first predetermined action is performed by the player character 9 (or a user), the determination unit 28 is activated. Then, when the player character 9 (or a user) performs a second predetermined action, the determination unit 28 recognizes that the action is performed on the image displayed on the screen object 5. Since the specific content of the predetermined action varies depending on the type and content of the game, the determination unit 28 determines the content of the first predetermined action and the second predetermined action according to the content of the target area 52. Alternatively, when only a specific type of game is executed, the content of the first predetermined action and the second predetermined action may be determined in advance.

[0042] In the case of a shooting game, the first predetermined action is, for example, an action of the player character 9 aiming a gun, and the second predetermined action is, for example, an action of operating a trigger (an action of firing a bullet from a gun). Alternatively, in the case of a dance game, the first predetermined action is an action of the player character 9 facing the screen object 5, and the second predetermined action is any action that changes the posture from a state facing the screen object 5. In the case of a dance game consisting of two phases, an example phase and a play phase, the determination unit 28 detects the first predetermined action and the second predetermined action only in the play phase. Alternatively, the user may notify the virtual space generating device 10 of the timing at which the player character 9 performs a predetermined action on the image (the timing at which the action is started) by a predetermined input operation on the user terminal 40, and the determination unit 28 may operate for a predetermined time (for example, 10 seconds) in response to the notification.

[0043] When the determination unit 28 recognizes that the action of the player character 9 has been performed on the image displayed on the screen object 5, the determination unit 28 determines whether or not the action has been performed on the target area 52. For example, in a shooting game, the determination unit 28 determines whether or not a bullet fired from a gun held by the player character 9 passes through the target area 52. In a game in which some object is fired from the player character 9 toward the screen object 5, such as a shooting game, the effect generation unit 29 fires a ray indicating a bullet trajectory 91 from the muzzle of the gun held by the player character 9 in the direction in which the barrel extends.

[0044] At this time, the determination unit 28 virtually places a transparent determination panel 51 on the back side of the screen object 5 (the side opposite to the player character 9), and determines whether or not the bullet has hit the target area 52 based on whether or not there is an overlap between an incident position 512 on the determination panel 51 of a light ray that has passed through the screen object 5 and a target area 511 on the determination panel 51 that corresponds to the target area 52 on the screen object 5 (see FIG. 4). This is an example of hit determination by the determination unit 28, and the method of hit determination can be changed as appropriate.

[0045] Also, for example, in the case of a dance game, as shown in FIG. 5, the determination unit 28 can be configured to virtually place a light source 54 behind the player character 9 (opposite the screen object 5), virtually place a transparent determination panel 51 behind the screen object 5, and score the dance performed by the player character 9 based on the degree of overlap between a silhouette area 513 of the player character 9 projected on the determination panel 51 by irradiating the player character 9 with light from the light source 54 and a target area 511 on the determination panel 51 corresponding to the target area 52 in the screen object 5 (see FIG. 6). It is not essential to use the light source 54, and any configuration can be adopted as long as it can project the silhouette area 513 of the player character 9 on the determination panel 51. For example, a camera that captures the player character 9 from the side of the screen object 5 may be virtually placed, and the silhouette captured by the camera may be projected on the determination panel 51. As described above, if the physique of a model performing an exemplary movement in a dance game does not match that of the player character, the determination result of the degree of match of the movement may be poor. Therefore, if the image data for the dance game has not been generated based on input information regarding the gender and physique of the player character, it is advisable to carry out processing such as aligning the physiques (height and body width) of both characters when projecting the silhouette of the player character 9 on the judgment panel 51.

[0046] Here, the dance performed by the player character 9 is scored based on the degree of overlap of the target area 511 on the judgment panel 51 corresponding to the target area 52 on the screen object 5, but various other methods can be adopted. For example, the posture acquisition unit 42 of the user terminal 42 may be configured to extract the user's bone data from the detected posture of the user, and the player character movement processing unit 23 may be configured to move the player character 9 based on the user's bone data, and the judgment unit 28 may extract bone data of a model performing an exemplary movement from the target area 52 and score the dance based on the degree of match with the user's bone data. If the judgment unit 28 extracts the model's bone data from the target area 52 in real time while playing the dance game and judges the degree of match with the user's bone data, the processing load on the judgment unit 28 may become excessive. Therefore, when the judgment unit 28 makes a judgment based on the degree of match between the bone data of the user and the model, it is preferable to perform a process of extracting the model's bone data from the image data in advance after the image data is selected or generated and before the game using the image data is started. Alternatively, in the case of a dance game consisting of two phases, an example phase and a play phase, bone data of the model may be extracted while image data of the example phase is being played back.

[0047] Alternatively, the posture acquisition unit 42 may be configured to extract information on changes in the position of a controller held in the user's hand or a specific object worn on the user's foot, and the player character movement processing unit 23 may be configured to reflect this position information in the movement of the player character 9 (changes in the position of the limbs), and the judgment unit 28 may extract changes in the position of the model's limbs from the target area 52 and score the dance based on the degree of match with the changes in the position of the user's limbs (in other words, the dance may be scored based on the degree of match between the position of a specific part of the user and the change in the position of the specific part of the model included in the target area 52).

[0048] The result of the determination by the determination unit 28 is displayed as an image (or video) on the screen object 5 by the effect generation unit 29. For example, in the case of a shooting game, as shown in FIG. 2, the effect generation unit 29 can be configured to display an effect 53 representing an explosion or an explosion around the impact position in the target area, or to display an effect of blood splashing from the impact position. Also, a sound effect indicating that the bullet has hit the target area 52 may be generated as an effect. Alternatively, the word "HIT!" can be displayed near the impact position in the target area 52 to indicate that the bullet has hit the target area. Also, in the case of a dance game, as shown in FIG. 5, the effect generation unit 29 can be configured to display the result of the determination by the determination unit 28 as a score on the screen object 5, or to display words such as "Excellent!", "Good!", and "Bad" according to the result of the determination by the determination unit 28 at each time point. Data of these effects are stored in the effect data storage unit 114 in advance according to the type of game expected.

[0049] When the image (game video) used in the above game is newly generated by the image generator, after the game ends, the data of the image (game video) is stored in the image data storage unit 117 together with information of the target area 52 identified by the image classifier from the game video. Also, the display data of the virtual space 8 generated by the image classifier is stored in the virtual space display data storage unit 112 in association with the data of the game video. Furthermore, even when the game video used in the game is stored in the image data storage unit 117 in advance, after the game ends, the information of the target area 52 identified by the image classifier is stored in the image data storage unit 117, and the display data of the virtual space 8 generated by the image classifier is stored in the virtual space display data storage unit 112.

[0050] In the game system 1 of this embodiment, an image input from outside to the virtual space generating device 10 as described above is displayed on the screen object 5, and the game can be enjoyed using the image. Although the use of screen objects has been proposed in the past, it has been limited to passively enjoying the image displayed there. In contrast, by using the virtual space generating device 10 of this embodiment and the game system 1 including the virtual space generating device 10, a game that has not been seen before can be enjoyed using the image displayed on the screen object.

[0051] Furthermore, in the above embodiment, the user can input words or sentences by voice or text to generate images (game images) according to his / her own tastes and enjoy them.

[0052] The above embodiment is merely an example and can be modified as appropriate in accordance with the spirit of the present invention.

[0053] Although the above embodiment has been described as a game system 1, the virtual space generation device according to the present invention can be used for purposes other than games. For example, a system having a similar configuration to the above embodiment can be suitably used for practicing yoga, training, and other training. In addition, in the above embodiment, the game system 1 is configured by the virtual space generation device 10 and the user terminal 40, but a part or all of the memory unit and functional blocks of the virtual space generation device 10 can be stored in the memory unit of the user terminal 40. or a functional block.

[0054] In the above embodiment, an image including the target area is displayed on the screen object, but other methods can be adopted. For example, a hologram of an image corresponding to the target area may be generated in a virtual space without using a screen object. In addition, in the above embodiment, a rectangular screen object 5 is arranged and an image including the target area and the non-target area is displayed, but only the target area may be displayed on a screen object having the same outer shape as the target area. Since the screen object itself is arranged virtually, its shape can be changed instantly and arbitrarily. In the above embodiment, an example in which the determination unit 28 uses the determination panel 51 has been described, but the determination may be made without using the determination panel 51.

[0055] In the above embodiment, an example was described in which a game image was created by an image generator, and a target area included in the game image was detected by an image classifier. However, the image generator may generate an initial image including a target area, and the image generator may generate a game image by giving a predetermined movement to the target area. For example, in the case of an animal, a trajectory for charging toward the player character 9 may be set in advance, and in the case of an airplane, a trajectory for turning while heading toward the player character 9 may be set in advance. In this case, the image classifier can easily track the target area included in the game image using the movement data given by the image generator. It may also be configured to launch a predetermined attack from the target area toward the player character according to the characteristics of the target area (airplane, animal, etc.).

[0056] In the above embodiment, an example in which one user enjoys a game has been described, but a plurality of users can use their own user terminals 40 and simultaneously enjoy a game in one virtual space.

[0057] [Aspects] It will be apparent to those skilled in the art that the above-described exemplary embodiments are illustrative of the following aspects.

[0058] (Section 1) A virtual space generating device according to one aspect of the present invention comprises: a display data storage unit in which display data of a virtual space and display data of a player character operating in the virtual space are stored; a virtual space generation unit that generates a virtual space using the display data of the virtual space stored in the display data storage unit; a player character operation processing unit that uses the display data of the player character stored in the display data storage unit to cause the player character to operate in the virtual space in response to an external input operation; an image display processing unit that generates an image including a target area in the virtual space based on data of the image; a determination unit that, when a predetermined action is performed by the player character on the image, determines whether the action is performed on the target area; a judgment result output unit that outputs a judgment result by the judgment unit; Equipped with.

[0059] In the virtual space generating device according to paragraph 1, an image display processing unit generates an image including a target area in a virtual space. When a user performs an input operation to the virtual space generating device using an external input device to move a player character, a player character movement processing unit moves the player character in the virtual space in response to the input operation. The external input device is, for example, a head mounted display (HMD), and an image of the virtual space captured from the viewpoint of the player character is displayed on the display unit.

[0060] Data of an image to be generated in a virtual space is input from the outside. Information for identifying the position of the target area in the image may or may not be included in the data of the image input from the outside (i.e., the target area included in the image may be undefined at the time when the image display processing unit generates the image). In the latter case, the target area may be identified using a determination unit including a classifier using a trained model that has been machine-learned to identify target areas included in game images of various types of games. In addition, this image data may be prepared in advance, or may be generated by a large-scale language model (LLM) or the like based on keywords input by the user. In the latter case, the determination unit may also include a classifier using an LLM.

[0061] When a player character operated by a user performs a predetermined action on an image generated in a virtual space, the determination unit determines whether or not the action is performed on a target area, and the determination result output unit outputs the determination result. For example, if the generated image is related to a shooting game, the predetermined action is a shooting action by the player character, and the determination unit determines whether or not a bullet fired by the action of the player character hits a target area (enemy character, airplane, animal, etc.) included in the image. Alternatively, for example, if the image generated in a virtual space is related to a dance game, the predetermined action is a dance action by the player character, and the determination unit determines the degree of agreement between the body movement of the player character and the target area (the dancer's body) included in the image. This allows the user to enjoy an unprecedented game in which the player character moves on an image generated in a virtual space.

[0062] (Section 2) The virtual space generating device according to paragraph 2 is the virtual space generating device according to paragraph 1, The image display processing unit places a screen object in the virtual space and displays the image on the screen object. It is something.

[0063] (Section 3) The virtual space generating device according to paragraph 3 is the virtual space generating device according to paragraph 2, the screen object is a transparent object, The determination unit places a determination panel behind the screen object, and makes the determination based on an overlap between a range of an action performed by the player character on the image on the determination panel and the target area. It is something.

[0064] In the virtual space generating device according to paragraph 1, one method of generating an image in the virtual space is to display the image on a screen object arranged in the virtual space, as described in paragraph 2. In addition, in the virtual space generating device according to paragraph 2, as described in paragraph 3, it is possible to determine whether or not the action of the player character has been performed in the target area using a determination panel arranged behind the screen object.

[0065] (Section 4) The virtual space generating device according to paragraph 4 is a virtual space generating device according to any one of paragraphs 1 to 3, further comprising: an image generator storage unit storing an image generator for generating an image based on a word or sentence; an image information input receiving unit that receives input of words or sentences related to the image; Equipped with The image display processing unit generates data of the image by inputting the words or sentences inputted to the image information input receiving unit to the image generator. It is something.

[0066] In the virtual space generating device according to paragraph 4, a user can generate any image according to his / her own taste and enjoy a game using that image.

[0067] (Section 5) The virtual space generating device according to paragraph 5 is the virtual space generating device according to paragraph 4, The image generator includes a large-scale language model.

[0068] As an image generator in the virtual space generating device according to the fourth aspect, the large-scale language models (LLM) described in the fifth aspect can be suitably used.

[0069] (Section 6) The virtual space generating device according to paragraph 6 is a virtual space generating device according to any one of paragraphs 1 to 5, further comprising: An image classifier storage unit in which an image classifier for identifying an object included in an image is stored, the image classifier being configured with a trained model created by machine learning using training data consisting of a game image and a pair of target regions included in the game image. Equipped with The determination unit identifies a target region included in the image by the image classifier. It is something.

[0070] In the virtual space generating device according to paragraph 6, the target area is not specified in advance, and any image can be used to enjoy the game.

[0071] (Section 7) The virtual space generating device according to paragraph 7 is a virtual space generating device according to paragraph 6, The image classifier includes at least one of an object detection algorithm, an algorithm for estimating a trajectory of a moving object detected by the object detection algorithm, and a large-scale language model.

[0072] As an image classifier in the virtual space generating device according to paragraph 6, the object detection algorithm described in paragraph 7 (e.g., YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector)), an algorithm for estimating the trajectory of a moving object detected by the object detection algorithm (e.g., DeepSORT, StrongSORT), and / or large language models (LLM) can be suitably used. [Explanation of symbols]

[0073] 1. Game System 10...Virtual space generation device 11...Storage section 111...user information storage unit 112...Virtual space display data storage unit 113…Character data storage unit 114…Effect data storage section 115...Image generator memory unit 116...Image classifier memory unit 117...Image data storage unit 21...User authentication section 22...Virtual space generation section 23...Player character movement processing unit 24...Image information input reception section 25...Image generation section 26...Image recognition unit 27...Image display processing unit 28…Judgment section 29…Effect generation section 40...User terminal 41...Storage section 42…Attitude acquisition part 43...Audio input section 44…Audio output section 45...Input section 46…Display section 48...Data processing unit 5. Screen Object 51…Judgment Panel 511…Target area on the judgment panel 512…Incidence position 513…Player character silhouette area 52…Target area 53...Effects 54...Light source 6, 60...empty 61…sun 62, 621...cloud 7, 70...ground 71, 711…house 72, 721…road 8. Virtual space 9. Player character 91…Trajectory

Claims

1. a display data storage unit in which display data of a virtual space and display data of a player character operating in the virtual space are stored; a virtual space generation unit that generates a virtual space using the display data of the virtual space stored in the display data storage unit; a player character operation processing unit that uses the display data of the player character stored in the display data storage unit to cause the player character to operate in the virtual space in response to an external input operation; an image display processing unit that generates an image including a target area in the virtual space based on data of the image; a determination unit that, when a predetermined action is performed by the player character on the image, determines whether the action is performed on the target area; a judgment result output unit that outputs a judgment result by the judgment unit; Equipped with The image display processing unit places a screen object in the virtual space and displays the image on the screen object. A virtual space generating device comprising:

2. the screen object is a transparent object, The determination unit places a determination panel behind the screen object, and makes the determination based on an overlap between a range of an action performed by the player character on the image on the determination panel and the target area.

2. The virtual space generating device according to claim 1 .

3. moreover, an image generator storage unit storing an image generator for generating an image based on a word or sentence; an image information input receiving unit that receives input of words or sentences related to the image; Equipped with The image display processing unit generates data of the image by inputting the words or sentences inputted to the image information input receiving unit to the image generator.

2. The virtual space generating device according to claim 1 .

4. The image generator includes a large-scale language model.

4. The virtual space generating device according to claim 3.

5. moreover, An image classifier storage unit in which an image classifier for identifying an object included in an image is stored, the image classifier being configured with a trained model created by machine learning using training data consisting of a game image and a pair of target regions included in the game image. Equipped with The determination unit identifies a target region included in the image by the image classifier.

2. The virtual space generating device according to claim 1 .

6. The image classifier includes at least one of an object detection algorithm, an algorithm for estimating the trajectory of a moving object detected by the object detection algorithm, and a large-scale language model.

6. The virtual space generating device according to claim 5.

Images