Information processing system

Abstract

In order to achieve the above purpose, an information processing system according to one embodiment of the present technology is used for generating AR content, the information processing system comprising: a display device including a display unit that is capable of displaying AR content to a user, an imaging unit that is capable of imaging the environment surrounding the user, and a transmission unit that transmits a prompt for generating content to be displayed in the AR content; and a content generation device including a content generation unit that generates content on the basis of the prompt, and a content arrangement unit that, on the basis of a real image, arranges the content within the real image. The display device furthermore includes an AR content generation unit that generates the AR content on the basis of a composite image in which the content is arranged within the real image by the content arrangement unit.

Description

Information processing system

[0005]

[0001] This technology relates to an information processing system in AR (Augmented Reality) content.

[0002] As a conventional AR (Augmented Reality) application, there is a management device that stores a character assigned to a target person, an original image in which a range equivalent to the real visual field of an observer is reflected, and at least a part of the target person in the original image. An AR unit that virtually transforms the target person by providing a composite image in which a part of the image of the character assigned to the target person is synthesized to the observer as the virtual visual field of the observer, and the management device changes the type of character candidate according to at least one of the action of the target person, the position of the target person, the current weather, and the current date and time. A virtual transformation system is disclosed (Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-96063

[0004] However, in the technology described in Patent Document 1, in addition to the user getting bored because pre-determined content is displayed, there is a problem that the content to be displayed is displayed at a position that is uncomfortable for the user.

[0005] In view of the above circumstances, an object of the present technology is to provide an information processing system that provides more content to the user and reduces the situation where the content is arranged in a way that is uncomfortable for the user.

[0006] An information processing system according to one embodiment of this technology is an information processing system for generating Augmented Reality (AR) content, comprising a display device and a content generation device. The display device has a display unit capable of displaying the AR content to a user, an imaging unit capable of capturing the user's surrounding environment, and a transmission unit that transmits prompts for generating content to be displayed on the AR content. The content generation device has an acquisition unit that acquires a real-world image of the user's surrounding environment captured by the imaging unit and the prompt, a content generation unit that generates the content based on the prompt, and a content placement unit that places the content within the real-world image based on the real-world image. The display device further has an AR content generation unit that generates the AR content based on a composite image obtained by placing the content within the real-world image by the content placement unit.

[0007] The content generation unit described above may generate the content based on the real-world image described above.

[0008] The content placement section described above may place the content in a natural position within the real-world image described above.

[0009] The above-mentioned display device further includes an image generation unit that generates a position-adjusted image in which the above-mentioned content is placed on the above-mentioned real-world image, and a comparison unit that compares the position-adjusted image generated by the image generation unit with the composite image, and the image generation unit may adjust the position of the above-mentioned content in the above-mentioned real-world image based on the comparison result of the comparison unit.

[0010] The above-mentioned content generation unit and content placement unit may be AI-generated.

[0011] This is a block diagram of an information processing system according to the first embodiment of this technology. This is a diagram showing the display device of the information processing system according to the first embodiment of this technology. This is a diagram showing the arrangement of the content of the information processing system according to the first embodiment of this technology. This is a diagram showing the sequence of the information processing system according to the first embodiment of this technology.

[0012] The embodiments of this technology will be described below with reference to the drawings.

[0013] <First Embodiment> Figure 1 is a block diagram of the information processing system 1 according to the first embodiment of this technology, and Figure 2 is a diagram showing the display device 10 of the information processing system 1 according to the first embodiment of this technology. Furthermore, Figure 3 is a diagram showing the arrangement of the content C of the information processing system 1 according to the first embodiment of this technology.

[0014] Information processing system 1 is a system for generating AR (Augmented Reality) content, and comprises a display device 10 and a content generation device 20. In this embodiment, pets (companion animals) are used as an example of content, but of course, it is not limited to this, and may also be vehicles such as airplanes or anime characters. In this embodiment, AR content is described as content superimposed on a real-world image, which will be described later.

[0015] (Display Device) The display device 10 includes a first acquisition unit 11, an instruction generation unit 12, an image generation unit 13, a comparison unit 14, an AR generation unit 15, a transmission unit 16, a first storage unit 17, an imaging unit 18, and a display unit 19, and is configured to communicate with a content generation device 20, which will be described later. The display device 10 is configured to display a virtual image (AR content) on the display (display unit 19) of a smartphone held in the user's hand. In this embodiment, the display device 10 is described using a smartphone as an example, but it is not limited to this, and may be a head-mounted display worn on the user's head.

[0016] As shown in Figure 2, the display unit 19 is a smartphone display (for example, composed of an LCD or OLED) and is configured to display images of the real world (the user's surrounding environment) captured by the imaging unit 18, which will be described later.

[0017] As shown in Figure 3, the display unit 19 is configured to provide (display) AR content to the user, in which a predetermined image (content) P is placed (superimposed) on the aforementioned video (real image) G of the real space.

[0018] The imaging unit 18 includes components necessary for shooting, such as a lens and an image sensor, and is installed in a part of the smartphone's casing. The image of the real world captured by the imaging unit 18 is displayed on the display unit 19.

[0019] The first acquisition unit 11 acquires (downloads) the 3D content generated by the content generation device 20 (described later) to the display device 10. The first acquisition unit 11 also acquires (downloads) a composite image on the display device 10, which is obtained by superimposing the content onto a real image P generated by the content generation device 20 (described later).

[0020] The instruction generation unit 12 generates instruction information that instructs the content generation device 20 (described later) to generate content and composite images.

[0021] Instruction information (content generation information) includes, in the case of content generation, a real-world image captured by the imaging unit 18 and a prompt entered by the user via the input means. Furthermore, in the case of instruction information (content placement information) that instructs the generation of a composite image, it includes content generated by the content generation device 20, a real-world image, and a prompt. Here, a prompt is an instruction given to the generation AI, and in this embodiment, it is an instruction (text) to the image generation AI on what kind of image (content to be displayed in the AR content) to generate. In this embodiment, the user enters the prompt by tapping on the display unit 19, but of course, it is not limited to this, and voice input may also be used.

[0022] The image generation unit 13 generates a position adjustment image by placing (superimposing) a rendered content (2D image) obtained by rendering the 3D content generated by the content generation device 20 into the real image captured by the imaging unit 18. The image generation unit 13 renders the 3D content based on the content of the composite image. In other words, the image generation unit 13 generates a position adjustment image by placing a rendered content, which has had its coordinates, rotation, and size adjusted using the composite image (the position of its content) as the true value.

[0023] The comparison unit 14 compares the position-adjusted image generated by the image generation unit 13 with the composite image generated by the content generation device 20, which will be described later. The comparison unit 14 compares the pixel values ​​of the two images. In this embodiment, pixel values ​​are used as an example, but of course, it is not limited to this, and other means that can quantify the similarity between the two images may be used, such as histogram comparison, edge comparison, or comparison of the coordinates of objects obtained by object recognition.

[0024] Based on the comparison results from the comparison unit 14, the image generation unit 13 generates a position-adjusted image again if the difference in pixel values ​​between the composite image (the position, size, and orientation of the content) and the position-adjusted image (the position, size, and orientation of the content) is greater than or equal to a predetermined threshold. In other words, the image generation unit 13 generates a position-adjusted image that approaches the composite image (the position, size, and orientation of the content). The image generation unit 13 terminates image generation if the comparison results (difference in pixel values) from the comparison unit 14 are less than a predetermined threshold, or if a predetermined number of adjustments have been made.

[0025] The AR generation unit 15 generates AR content (AR) by superimposing rendering content P, which is 3D content rendered onto a real image G captured by the imaging unit 18, based on the adjustment results of the image generation unit 13 (see Figure 3). The generated AR content (AR) is displayed on the display unit 19.

[0026] The transmission unit 16 transmits the instruction information generated by the instruction generation unit 12 to the content generation device 20, which will be described later. The transmission unit 16 transmits the instruction information to the content generation device 20 by wire or wireless connection.

[0027] The first storage unit 17 is implemented by a storage device such as a semiconductor memory element, such as RAM (Random Access Memory). The first storage unit 17 stores 3D content and composite images downloaded from the content generation device 20, as well as programs for various processes executed in the image generation unit 13 and the AR generation unit 15.

[0028] (Content generation device) The content generation device 20 includes a second acquisition unit 21 (acquisition unit), a content generation unit 22, a content placement unit 23, and a second storage unit 24.

[0029] The second acquisition unit 21 acquires the instruction information transmitted from the transmission unit 16 described above.

[0030] The content generation unit 22 generates content (3D content) based on the real-world image and prompt included in the instruction information (content generation information). The content generation unit 22 uses an image generation AI model to generate images (content) corresponding to the generated text (prompt) and real-world image. Examples of AI automatic image generation services using an image generation AI model include Canva, PhotoDirector, and Midjourney.

[0031] As shown in Figure 2, the text content can be, for example, "hearty appetite," "energetic," "cute appearance," or "squirrel," and an image of a squirrel will be generated, as shown in Figure 3. A weight can be assigned to each text. In other words, in the case of the text mentioned above, if a weight is assigned to "hearty appetite," an image of a squirrel with its cheeks full of food will be generated.

[0032] As mentioned above, content is generated not only from text but also from real-world images. As shown in Figure 3, the real-world image is of an indoor setting. This indicates that the squirrel is kept as a pet. The types of squirrels that can be kept as pets are chipmunks, Richardson's ground squirrels, black-tailed prairie dogs, Columbian ground squirrels, and banana squirrels, and an image of one of these squirrels will be generated. If the real-world image is of a forest, an image of a squirrel species that can live in that forest will be generated.

[0033] The content placement unit 23 determines the position of the content to be placed within the real image based on the real image. Specifically, the content placement unit 23 generates a composite image in which the content is placed (superimposed) on the real image based on the content included in the instruction information (content placement information), the real image, and the prompt. The content placement unit 23 also generates the composite image using an image generation AI model, similar to the content generation unit 22.

[0034] When the content placement unit 23 places content within a real-world image, it considers the image information and prompt conditions within the real-world image to place the content in a natural position within the real-world image.

[0035] Placing content in a natural position means that it would look natural (or have minimal impact) if it were placed within a real-world image. For example, in Figure 2, the squirrel is placed near the cookie in the real-world image because it is healthy and has a hearty appetite. This is based on the assumption that a healthy and hungry squirrel would be interested in the cookie, which is food, and therefore would be in a nearby location.

[0036] In other words, the content placement unit 23 places the content in a location that seems most likely to be where it should be placed within the real image, based on the image information within the real image (what is present) derived from the characteristics of the content (what the content is (what kind of creature it is, etc.), and the conditions for the content specified when it was generated (energetic, with a good appetite, etc.)).

[0037] In other words, the content placement unit 23 detects common or related information (in this embodiment, appetite and cookies) from the characteristics of the content and the image information in the real image, and places the content in the real image based on that common or related information (i.e., places the content near the cookies).

[0038] The second storage unit 24 is implemented by a storage device such as a semiconductor memory element, such as RAM (Random Access Memory). The second storage unit 24 stores programs for various processes that are executed when generating the 3D content and composite images mentioned above.

[0039] (Method for generating AR content) Figure 4 is a sequence diagram of the information processing system 1 according to the first embodiment of this technology. Here, the method for generating AR content will be explained using Figure 4. As a premise, in this embodiment, a pet squirrel is displayed as AR content.

[0040] First, the display device 10 transmits content generation information to the content generation device 20 (S101). The instruction generation unit 12 of the display device 10 generates instruction information (content generation information) that includes a real image captured by the imaging unit 18 and a prompt entered by the user via the input means. Then, the transmission unit 16 transmits the generated content generation information to the content generation device 20.

[0041] Next, the content generation device 20 generates content (3D content) (S102). The content generation unit 22 of the content generation device 20 generates content based on the content generation information transmitted by the transmission unit 16 described above. In this embodiment, the content is a squirrel.

[0042] Next, the content is downloaded from the content generation device 20 (S103), and the user checks whether there are any problems with the downloaded content (S104). If the user checks the content and determines that there are no problems, the process proceeds to the next step. If the user determines that there are problems (for example, if the appearance is unnatural or the user does not like it), the user may be instructed to generate the content again (repeat S101).

[0043] Next, the display device 10 transmits content placement information to the content generation device 20 (S105). If the user confirms the content and determines that there are no problems, the instruction generation unit 12 of the display device 10 generates instruction information (content placement information) that includes the content generated by the content generation device 20, a real image, and a prompt. The transmission unit 16 then transmits the generated content placement information to the content generation device 20.

[0044] Next, the content generation device 20 generates a composite image (S106). The content placement unit 23 of the content generation device 20 generates a composite image in which content is placed (superimposed) on the real image based on the above-described content placement information. In the present embodiment, a list is placed near the cookie C in the real image.

[0045] Next, the user downloads the composite image from the content generation device 20 (S107) and checks whether there is any problem with the downloaded composite image (S108). If the user checks the composite image and determines that there is no problem, the process proceeds to the next step. If it is determined that there is a problem (for example, there is a sense of incongruity in the placement of the content (the position of the list and the cookie is too close or too far)), the instruction information may be generated again, and the content generation device 20 may be instructed to generate the composite image again (perform S105 again).

[0046] Next, the display device 10 compares the position adjustment image with the composite image (S109). If the user checks the composite image and determines that there is no problem, the comparison unit 14 of the display device 10 compares the pixel values of the position adjustment image (the position, size, and orientation of the content) with the pixel values of the composite image (the position, size, and orientation of the content). If the difference in the pixel values of the two images is less than a predetermined threshold, the process proceeds to the next step. If the difference in the pixel values of the two images is greater than or equal to the predetermined threshold, the image generation unit 13 adjusts until the difference is less than the predetermined threshold. The number of adjustments may be determined, for example, to be 5 times, etc., and when the predetermined number of times is performed, the process may proceed to the next step.

[0047] Next, the display device 10 generates AR content AR (S110). The AR generation unit 15 of the display device 10 generates AR content AR, which is an extended reality image in which the rendered content P is placed (superimposed) on the real image G like a composite image, based on the adjustment result of the image generation unit 13 (the position adjustment image that has become less than the predetermined threshold or, for example, the last generated position adjustment image when the predetermined number of times is performed) (see FIG. 3). The generated AR content AR is displayed on the display unit 19.

[0048] As described above, in the present embodiment, the information processing system 1 determines the location where the generated content is to be placed based on the real image. Thereby, it is possible to reduce the likelihood that the content will be placed in a location that feels unnatural to the user. That is, conventionally, since the generated content was simply placed on the camera image (real image), the user may have felt a sense of discomfort at the location where the content was placed (for example, a hungry lizard turning its back on a cookie, etc.). However, in the present embodiment, as described above, since the content is configured to be placed in a natural position, it is possible to reduce the user's sense of discomfort.

[0049] Also, in the present embodiment, since the content generation unit 22 is an image generation AI, it is possible to provide the user with a greater number (an infinite number) of contents, and it is possible to reduce the likelihood that the user will become bored with the AR content (conventionally, since the pre-determined content was displayed and the patterns were limited).

[0050] Furthermore, in the present embodiment, since the 3D content is first generated and the position of the content is adjusted, when the user uses the AR content, the computational cost can be reduced to only the content position estimation algorithm compared to the case where the AR content is generated for each frame.

[0051] Also, in the present embodiment, the number of real images captured by the imaging unit 18 is not particularly limited, but it is preferably 2 or more. That is, based on two or more real images with different viewpoints, composite images are generated respectively, thereby improving the adjustment accuracy of the image generation unit 13 and the accuracy of the position of the content arranged by the AR generation unit 15.

[0052] Each configuration of the information processing system described with reference to each drawing is merely an embodiment, and can be arbitrarily modified without departing from the gist of the present technology. That is, any other arbitrary configuration for implementing the present technology may be adopted.

[0053] Furthermore, this technology can also take the following configurations: (1) An information processing system for generating AR (Augmented Reality) content, comprising: a display device having a display unit capable of displaying the AR content to a user; an imaging unit capable of imaging the user's surrounding environment; and a transmission unit that transmits a prompt for generating content to be displayed on the AR content; and a content generation device having a second acquisition unit that acquires a real image of the user's surrounding environment captured by the imaging unit and the prompt; a content generation unit that generates the content based on the prompt; and a content placement unit that places the content within the real image based on the real image, wherein the display device further comprises a first acquisition unit that acquires a composite image in which the content is placed within the real image by the content placement unit; and an AR content generation unit that generates the AR content based on the composite image acquired by the first acquisition unit. (2) The information processing system according to (1) above, wherein the content generation unit generates the content based on the real image. (3) An information processing system according to (1) or (2) above, wherein the content placement unit is an information processing system that places the content in a natural position within the real image. (4) An information processing system according to any one of (1) to (3) above, wherein the display device further comprises an image generation unit that generates a position adjustment image in which the content is placed in the real image based on the composite image, and a comparison unit that compares the position adjustment image generated by the image generation unit with the composite image, wherein the image generation unit adjusts the position of the content in the real image based on the comparison result of the comparison unit. (5) An information processing system according to any one of (1) to (4) above, wherein the generated content unit and the content placement unit are generated AI.

[0054] 1... Information processing system 10... Display device 11... First acquisition unit 13... Image generation unit 15... AR generation unit 18... Imaging unit 19... Display unit 20... Content generation device 21... Second acquisition unit 22... Content generation unit 23... Content placement unit AR... AR content

Claims

1. An information processing system for generating Augmented Reality (AR) content, comprising: a display device having a display unit capable of displaying the AR content to a user; an imaging unit capable of imaging the user's surrounding environment; and a transmission unit that transmits a prompt for generating content to be displayed on the AR content; and a content generation device having an acquisition unit that acquires a real image of the user's surrounding environment captured by the imaging unit and the prompt; a content generation unit that generates the content based on the prompt; and a content placement unit that places the content within the real image based on the real image, wherein the display device further comprises an AR content generation unit that generates the AR content based on a composite image obtained by the content placement unit placing the content within the real image.

2. An information processing system according to claim 1, wherein the content generation unit generates the content based on the real image.

3. An information processing system according to claim 1, wherein the content placement unit places the content in a natural position within the real image.

4. An information processing system according to claim 1, wherein the display device further comprises an image generation unit that generates a position adjustment image in which the content is placed on the real image, and a comparison unit that compares the position adjustment image generated by the image generation unit with the composite image, and the image generation unit adjusts the position of the content in the real image based on the comparison result of the comparison unit.

5. An information processing system according to claim 1, wherein the content generation unit and the content placement unit are a generating AI.

Images