Information processing device, method, and program

Abstract

The invention provides an information processing device, method, and program capable of providing a composite space or a virtual space with excellent interestingness. First information relating to an environment of a space in which an object appears is acquired. Second information relating to the behavior history of the user is acquired. A first element related to the object is identified on the basis of the first information. A second element related to the object is identified on the basis of the second information. An object appearing in the space is identified on the basis of the first element and the second element.

Description

Technical Field

[0001] The present invention relates to an information processing device, method and program, and in particular to an information processing device, method and program that provides a composite space or a virtual space that integrates the real world and the virtual world. Background Art

[0002] Augmented reality (AR) is a technology that displays a virtual world over the real world. AR expands the real world by displaying digital content such as characters created by computer graphics (CG) over real scenery.

[0003] Patent Document 1 describes a technique for creating an augmented reality image with characters superimposed thereon, which includes changing the decoration of the superimposed characters according to the usage status (intimacy) and usage environment (season, location, etc.) of the characters.

[0004] Furthermore, Patent Document 2 describes a technique for creating an augmented reality image with characters superimposed thereon, which describes a technique for determining a surrounding environment from a captured image and causing a character that matches the determined surrounding environment to appear.

[0005] Furthermore, Patent Document 3 describes a technique in which attributes of a user are estimated based on the behavior history of the user, and attributes of a character are selected based on the estimated attributes.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2018-36869

[0009] Patent Document 2: Japanese Patent Application Publication No. 2019-212062

[0010] Patent Document 3: International Publication No. 2016 / 111067 Summary of the invention

[0011] One embodiment of the technology according to the present invention provides an information processing device, method, and program that can provide a complex space or a virtual space with excellent interest.

[0012] Means for solving technical problems

[0013] (1) An information processing apparatus includes a processor that performs the following processes: acquiring first information related to the environment of the space where an object appears; acquiring second information related to the user's behavior history; determining a first factor related to the object based on the first information; determining a second factor related to the object based on the second information; and determining the object that appears in the space based on the first factor and the second factor.

[0014] (2) The information processing apparatus according to (1), wherein the processor performs the following processes: acquiring a first image that captures the space where the object appears; and generating a second image with the object superimposed on the first image.

[0015] (3) The information processing apparatus according to (2), wherein the processor processes the first image to identify an object within the first image or the scene of the first image, and acquires the first information.

[0016] (4) The information processing apparatus according to (3), wherein the processor performs the following processes: acquiring information on the user's position; and when the user is present in a specific area, identifying an object within the first image or the scene of the first image, and acquiring the first information.

[0017] (5) The information processing apparatus according to any one of (1) to (4), wherein the second information includes information on the user's movement history or information on the history related to the object that appears.

[0018] (6) The information processing apparatus according to any one of (1) to (5), wherein the first factor is the type of the object that appears, and the second factor is the display mode or attribute of the object that appears.

[0019] (7) The information processing apparatus according to any one of (1) to (6), wherein the processor performs the following processes: processing an image that captures the space where the object appears to identify the main subject; and determining the arrangement area of the object that appears in the space based on the position of the main subject within the space.

[0020] (8) The information processing apparatus according to (7), wherein the processor determines the type of the object that appears based on the first information within the arrangement area.

[0021] (9) The information processing apparatus according to any one of (6) to (8), wherein the processor randomly determines the display mode of the object when the user's behavior history meets specific conditions.

[0022] (10) The information processing apparatus according to any one of (2) to (4), wherein the processor changes the display mode or attribute of the object according to the output instruction of the second image.

[0023] (11) An information processing method, comprising the following steps: a step of acquiring first information related to the environment of the space where an object appears; a step of acquiring second information related to the user's behavior history; a step of determining a first element related to the object based on the first information; a step of determining a second element related to the object based on the second information; and a step of determining the object that appears in the space based on the first element and the second element.

[0024] (12) An information processing program that causes a computer to implement the following functions: a function of acquiring first information related to the environment of the space where an object appears; a function of acquiring second information related to the user's behavior history; a function of determining a first element related to the object based on the first information; a function of determining a second element related to the object based on the second information; and a function of determining the object that appears in the space based on the first element and the second element. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a diagram showing an outline of information processing.

[0026] Figure 2 is a diagram showing an example of the hardware structure of a mobile terminal.

[0027] Figure 3 is a block diagram of the main functions of a mobile terminal for AR display.

[0028] Figure 4 is a block diagram of the main functions of an environmental information acquisition unit.

[0029] Figure 5 is a diagram showing an example of object recognition.

[0030] Figure 6 is a block diagram of the main functions of an AR object determination unit.

[0031] Figure 7 is a conceptual diagram of the determination of the arrangement area of an AR object.

[0032] Figure 8 is a diagram showing an example of the information recorded in an AR object database.

[0033] Figure 9 is a diagram showing an example of the display mode of an AR object.

[0034] Figure 10 is a diagram showing an example of an AR image.

[0035] Figure 11 is a flowchart showing the operation sequence of a mobile terminal during AR display.

[0036] Figure 12 It is a flowchart showing the determination processing order of the display method of the AR object.

[0037] Figure 13 It is a conceptual diagram of AR image generation.

[0038] Figure 14 It is a flowchart showing an example of the processing order when conducting a lottery to determine the display method of the AR object.

[0039] Figure 15 It is a diagram showing an example of the lottery screen.

[0040] Figure 16 It is a block diagram of the main functions of the AR object determination unit.

[0041] Figure 17 It is a diagram showing an example of the information recorded in the AR object database.

[0042] Figure 18 It is a diagram showing an example of the appearance of the character.

[0043] Figure 19 It is a flowchart showing the order of the determination process of the appearance of the character.

[0044] Figure 20 It is a diagram showing an example of AR display.

[0045] Figure 21 It is a diagram showing an example when the character performs a specific action by shooting.

[0046] Figure 22 It is a diagram showing an example of the display during printing.

[0047] Figure 23 It is a diagram showing an example of the history data of the character attribute change.

[0048] Figure 24 It is a conceptual diagram showing an example of the configuration of the character.

[0049] Figure 25 It is a conceptual diagram showing an example of the configuration of the character using image recognition.

[0050] Figure 26 It is a conceptual diagram of printing. Detailed implementation mode

[0051] Hereinafter, the preferred implementation modes of the present invention will be described with reference to the accompanying drawings.

[0052] [First implementation mode]

[0053] Figure 1 It is a diagram showing the outline of the information processing of this implementation mode.

[0054] like Figure 1 As shown, the information processing of this embodiment uses a mobile terminal 10 equipped with a camera and a display to provide a composite space that integrates the real world and the virtual world. Specifically, an augmented reality object (AR object) ARO produced by CG is superimposed on the image of the real space (real space, actually existing space) captured by the camera of the mobile terminal 10 to generate an augmented reality image (AR image), and it is displayed on the display 15 of the mobile terminal 10. The AR object ARO appears triggered by the user's position. At this time, its type, display method or attribute is determined according to the environment of the space in which it appears (the environment of the space captured by the camera) and the user's behavior history. In this way, a composite space with excellent fun can be provided.

[0055] [Structure of mobile terminal]

[0056] Figure 2 This is a diagram showing an example of the hardware structure of a mobile terminal.

[0057] Figure 2 The mobile terminal 10 shown is a so-called smart phone. The mobile terminal 10 includes a CPU (Central Processing Unit) 11 for controlling the overall operation, a ROM (Read Only Memory) 12 storing basic input and output programs, etc., a RAM (Random Access Memory) 13 used as a work area of ​​the CPU 11, a built-in memory 14, a display 15, a touch panel 16 for detecting touch operations (position input) on the display screen, a GPS receiving unit 17 for receiving GPS signals including position information (latitude, longitude and altitude) of the mobile terminal 10 via GPS (Global Positioning System) satellites or IMES (Indoor MEssaging System) as an indoor GPS, a camera unit 18 for electronically capturing images, a microphone unit 19 for inputting sounds via a microphone, a speaker unit 20 for outputting sounds via a speaker, a wireless communication unit 21 for wirelessly communicating with the nearest base station via an antenna, a short-distance wireless communication unit 22 for wirelessly communicating with an external device via an antenna, and a sensor unit 23 for performing various sensing. The built-in memory 14 is composed of a nonvolatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory).

[0058] In the built-in memory 14, in addition to storing various programs including the operating system (e.g., information processing programs, etc.), various data required for control and processing, image data captured by the internal camera unit 18A and the external camera unit 18B, etc. are also stored.

[0059] The display 15 is composed of, for example, an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diode) display, etc. The display 15 and the touchpad 16 together form the touch panel 30. The touch panel 30 constitutes the operation unit of the mobile terminal 10.

[0060] The camera unit 18 is composed of the internal camera unit 18A and the external camera unit 18B. The internal camera unit 18A is a camera mounted on the front side of the housing of the mobile terminal 10 (refer to Figure 1 ). The external camera unit 18B is a camera mounted on the back side of the housing of the mobile terminal 10. In addition, the front of the housing of the mobile terminal 10 refers to the surface of the housing of the mobile terminal 10 where the display 15 is provided. Therefore, the internal camera unit 18A is a camera mounted on the display 15 side. The internal camera unit 18A and the external camera unit 18B are digital cameras and have an optical system and an image sensor. The image sensor is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, etc. In addition, hereinafter, unless otherwise specifically distinguished, the image captured by the camera unit 18 refers to the image captured by either the internal camera unit 18A or the external camera unit 18B.

[0061] The wireless communication unit 21 performs wireless communication with a base station, for example, by means of a cellular network such as 5G (5th Generation Mobile Communication System), 4G (4th Generation Mobile Communication System), LTE (Long Term Evolution).

[0062] The short-range wireless communication unit 22 performs wireless communication with an external device, for example, by means of a short-range wireless communication method such as Wi-Fi (registered trademark), Bluetooth (registered trademark).

[0063] The sensor unit 23 includes various sensors such as an acceleration sensor, an angular velocity sensor (gyro sensor), a geomagnetic sensor, an ambient light sensor, a proximity sensor, LiDAR (Light Detection And Ranging), and a living body recognition sensor.

[0064] Figure 3 It is a block diagram of the main functions of a mobile terminal for AR display.

[0065] As Figure 3 shown, the mobile terminal 10 has functions such as an environmental information acquisition unit 111, a behavior history information acquisition unit 112, a behavior history database management unit 113 (database / Data Base; DB), an AR object determination unit 114, an AR image generation unit 115, a display control unit 116, and a recording control unit 117 for AR display. These functions are implemented by the CPU 11 executing a prescribed program (information processing program).

[0066] The environmental information acquisition unit 111 acquires information related to the environment of the space where the AR object appears. Information related to the environment of the space includes various information that constitutes or characterizes the space, such as geographical information of the space, objects that make up the space, and information about objects existing in the space. Information related to the environment of the space where the AR object appears is an example of the first information. In the present embodiment, since an AR image is generated by overlapping the AR object on the captured image, the objects included in the captured image are recognized to acquire information related to the environment of the space where the AR object appears. Also, information on the current position of the user (the position where the image is being captured) is acquired to obtain environmental information of the space where the AR object appears.

[0067] Figure 4 It is a block diagram of the main functions of the environmental information acquisition unit.

[0068] As Figure 4 shown, the environmental information acquisition unit 111 of the present embodiment has functions such as a position determination unit 111A and an object recognition unit 111B.

[0069] The position determination unit 111A determines the current position of the mobile terminal 10. The current position of the mobile terminal 10 is set as the current position of the user (the position where the image is being captured). The position determination unit 111A determines the current position of the mobile terminal 10 based on the GPS signal received by the GPS reception unit 17 and the map database 121.

[0070] In the map database 121, specific points of interest (POIs) on the map are recorded in association with their location information (latitude, longitude, and altitude). A point of interest is a concept that includes areas and regions. Also, points of interest can be determined by facilities such as parks, amusement parks, aquariums, art galleries, stores such as department stores and restaurants, scenic spots such as shrines and Buddhist temples, and tourist attractions.

[0071] The position determination unit 111A obtains information on the location information (latitude, longitude, and altitude) contained in the GPS signal, and searches the map database 121 for information on the point of interest corresponding to this location information, in order to obtain information on the current position of the mobile terminal 10. In the case where information on the point of interest corresponding to the obtained location information does not exist in the map database 121, the position determination unit 111A determines that the position is unknown. The determination result is output as the current position information of the mobile terminal 10.

[0072] The object recognition unit 111B analyzes the image captured by the camera unit 18 to identify the objects (objects) contained in the image. More specifically, object detection and detection of the area are performed. The objects here include the sky, the ground (including roads), etc. Well-known techniques are used to identify the objects. For example, a technique of object recognition (object recognition) using artificial intelligence (AI) can be utilized. As an example, a structure that uses a learned model that has been machine-learned in a manner to perform object recognition can be adopted. In this case, as the learned model, for example, a model using semantic segmentation, instance segmentation, panoptic segmentation, etc. can be adopted.

[0073] Figure 5 It is a diagram showing an example of object recognition.

[0074] Figure 5 (A) shows the captured image I (the image captured by the camera unit 18). Figure 5 (B) shows an example of the object recognition result of the captured image I. As Figure 5 (B) shows, the area of the object in the image is determined, and the area is segmented for each individual to identify the type of the object. In Figure 5 the example shown, a person (Human), grass (Grass), soil (Ground), a tree (Tree), and the sky (Sky) are recognized as objects from the captured image I.

[0075] The object recognition unit 111B can be configured to recognize only a predetermined specific object. For example, it can be configured to recognize only a person and the ground (such as grass, soil, artificial foundation, etc.).

[0076] When the current position of the mobile terminal 10 is a specific point of interest recorded in the map database 121, the object recognition unit 111B performs object recognition processing.

[0077] The behavior history information acquisition unit 112 acquires information related to the user's behavior history. The information related to the user's behavior history is an example of the second information. In the present embodiment, information on the location history is acquired as the user's behavior history. The location history refers to a history that records places, paths, etc. visited by the user while carrying the mobile terminal 10. The behavior history information acquisition unit 112 acquires information related to the user's behavior history from the behavior history database 122. In the present embodiment, information on the location history is acquired. Information related to the user's behavior history is recorded in the behavior history database 122. In the present embodiment, information on the location history is recorded.

[0078] The behavior history database management unit 113 manages the behavior history database 122. In the present embodiment, information on the location history is recorded in the behavior history database 122. Therefore, the behavior history database management unit 113 manages the information on the location history. The behavior history database management unit 113 acquires the location information of the mobile terminal 10 via the GPS reception unit 17 and manages the user's location history. That is, it manages histories such as places and paths visited by the user.

[0079] The AR object determination unit 114 determines the augmented reality object (AR object) that appears. The AR object determination unit 114 determines the AR object based on the information acquired by the environment information acquisition unit 111 and the information acquired by the behavior history information acquisition unit 112. In the present embodiment, the AR object is determined based on the information of the object (object information) in the captured image I acquired by the environment information acquisition unit 111 and the information on the location history (location history information) acquired by the behavior history information acquisition unit 112.

[0080] Figure 6 It is a block diagram of the main functions of the AR object determination unit.

[0081] As Figure 6 shown, the AR object determination unit 114 has functions such as a main subject determination unit 114A, a placement area determination unit 114B, a first element determination unit 114C, and a second element determination unit 114D.

[0082] The main subject determination unit 114A determines the main subject based on the information of the object (object information) in the captured image I. In the present embodiment, the person in the captured image I is set as the main subject. When there are multiple persons in the captured image I, the main subject determination unit 114A determines the main subject based on the position, size, etc. in the image. For example, the person closer to the center is recognized as the main subject. Also, the person with a larger size is recognized as the main subject.

[0083] The arrangement area determination unit 114B determines the arrangement area of the AR object based on the determination result of the main subject by the main subject determination unit 114A.

[0084] Figure 7 It is a conceptual diagram of the determination of the arrangement area of the AR object.

[0085] In Figure 7 the area indicated by the slashes is the arrangement area R of the AR object. In the present embodiment, the area around the main subject MO and on the ground is set as the arrangement area R of the AR object. Figure 7 It is an example when the area within a circle with a radius r centered on the main subject MO is set as the arrangement area R of the AR object. The radius r can be constant or can be set according to the size of the main subject MO and / or the distance of the main subject MO in the captured image I.

[0086] The first element determination unit 114C determines the type of the first element of the AR object that appears. In the present embodiment, the type of the AR object is determined based on the environmental information of the arrangement area R. More specifically, the type of the AR object is determined based on the information of the object recognized in the arrangement area R.

[0087] The first element determination unit 114C determines the type of the AR object by referring to the information recorded in the AR object database 123.

[0088] Figure 8 It is a diagram showing an example of the information recorded in the AR object database.

[0089] As Figure 8 shown, the information on the type and display method of the AR object displayed for each environment of the arrangement area is recorded. The information on the display method includes the data of the AR object required for display. Each AR object is constituted by three-dimensional CG (three-dimensional computer graphics; 3DCG) for example. Therefore, the data of the AR object is constituted by the data of three-dimensional CG.

[0090] The first element determination unit 114C selects, from among the types of multiple AR objects recorded in the AR object database 123, the type corresponding to the environment (recognized object) of the placement area R. For example, according to Figure 8 In the example of , when the environment of the placement area R is "soil", the type of the AR object that appears becomes "cat".

[0091] When multiple environments (recognized objects) are included in the placement area R, the environment with the highest occupancy rate is set as the environment of the placement area R. In Figure 7 In the example of , both "soil" and "grassland" are included in the placement area R, but the occupancy rate of "soil" is higher. Therefore, in Figure 7 In the example of , the environment of the placement area R is determined to be "soil".

[0092] The second element determination unit 114D determines the second element as the display mode of the AR object that appears. The second element determination unit 114D determines the display mode of the AR object according to the information related to the user's behavior history acquired by the behavior history information acquisition unit 112. In the present embodiment, the display mode of the AR object is determined according to the information of the location history. In the present embodiment, the display mode is determined according to whether it is the place where the user first visits. That is, the display mode is determined according to whether the current position of the mobile terminal 10 (the current position of the user) is the place existing in the location history. In the case of the place where the user first visits (the case of an unvisited location), the second element determination unit 114D sets the display mode of the AR object to the first display mode. This case is the case where the current position does not exist in the location history. On the other hand, in the case of the place that is not the user's first visit (the case of a visited location), the second element determination unit 114D sets the display mode of the AR object to the second display mode. This case is the case where the current position exists in the location history.

[0093] Figure 9 It is a diagram showing an example of the display mode of the AR object.

[0094] In the present embodiment, the number of the AR objects ARO that appear is set as the display mode of the AR object. The second display mode is set such that the number of the displayed AR objects ARO is more than that of the first display mode.

[0095] In addition, in Figure 9 , for the sake of convenience, each AR object ARO is two-dimensionally displayed, but each AR object ARO is composed of 3DCG. And, in Figure 9 , it is displayed in a stationary state, but each AR object ARO can also be set to a moving structure.

[0096] The AR image generation unit 115 generates an augmented reality image (AR image) in which an AR object is superimposed on the image captured by the camera unit 18 (captured image). The AR image generation unit 115 superimposes the AR object determined by the AR object determination unit 114 on the captured image I to generate an AR image. At this time, the AR image generation unit 115 configures the AR object in the configuration area determined by the configuration area determination unit 114B to generate an AR image. Further, the AR image generation unit 115 causes the AR object determined by the first element determination unit 114C to be displayed in the display mode determined by the second element determination unit 114D to generate an AR image (superimposing a three-dimensional CG in the display mode determined by the second element determination unit 114D to generate an AR image). In the present embodiment, the captured image is an example of the first image, and the AR image is an example of the second image. Further, the AR object is an example of an object.

[0097] Figure 10 This is a diagram showing an example of an AR image. Figure 10 (A) shows an example when displayed in the first display mode. Figure 10 (B) shows an example when displayed in the second display mode.

[0098] As Figure 10 As shown in (A) and (B), in the present embodiment, an AR image ARI in which the AR object ARO is superimposed on the periphery of the main subject MO and the ground area is generated.

[0099] The display control unit 116 controls the image display on the display 15. The display control unit 116 causes the AR image ARI generated by the AR image generation unit 115 to be displayed on the display 15 (see Figure 1 ).

[0100] The recording control unit 117 controls the image recording to the built-in memory 14. The recording control unit 117 records the image being displayed on the display 15 in the built-in memory 14 according to a recording instruction from the user (a still image and a moving image shooting instruction). The recording instruction is, for example, performed by a touch operation on a shutter button SSB displayed on the display 15 (see Figure 1 ).

[0101] [Operation of the mobile terminal (information processing method)]

[0102] Figure 11 This is a flowchart showing the operation sequence of the mobile terminal when performing AR display.

[0103] First, a live view image is displayed on the display 15 (step S10). Live view means that the image captured by the image sensor of the camera unit 18 is displayed in real time (immediately).

[0104] Next, information on the current position of the mobile terminal 10 (the current position of the user) is acquired (step S11). The mobile terminal 10 acquires information on the current position of the mobile terminal 10 based on the GPS signal received by the GPS receiving unit 17.

[0105] Next, based on the acquired information on the current position, it is determined whether the current position is a specific area set in the map database 121 (step S12). More specifically, it is determined whether the current position is a place existing in the map database 121.

[0106] In the case where the current position is the specific area determined in the map database 121, it is determined whether an AR object is displayed in the live view image being displayed on the display 15 (step S13).

[0107] In the case where the AR object is not displayed, a process of recognizing an object from the captured image is performed (step S14). Here, the "captured image" refers to the image displayed as the live view.

[0108] Based on the recognition result of the object, the main subject is determined from the captured image (step S15). In the present embodiment, a person is set as the main subject. Therefore, a person is determined as the main subject from the recognized objects. In the case where a plurality of persons are included in the image, the main subject is determined according to a predetermined determination criterion.

[0109] Based on the result of the main subject determination process, it is determined whether there is a main subject (step S16). That is, it is determined whether a main subject exists in the captured image.

[0110] In the case where a main subject exists in the captured image, the arrangement area of the AR object is determined based on the position of the main subject in the image (step S17). In the present embodiment, the area around the main subject and on the ground is set as the arrangement area of the AR object.

[0111] After determining the arrangement area of the AR object, environmental information of the arrangement area is acquired (step S18). In the present embodiment, information on the object in the arrangement area of the AR object is acquired based on the recognition result of the object.

[0112] Based on the acquired information on the object in the arrangement area, the AR object that appears in the image is selected (step S19). That is, it is determined which type of AR object appears. The AR object is selected from the AR object database 123.

[0113] If the appearing AR object is determined, information on the user's behavior history is acquired (step S20). In the present embodiment, information on the position history is acquired.

[0114] Determine the display mode of the AR object based on the information of the obtained location resume (step S21). In the present embodiment, determine the display mode of the AR object according to whether the current location of the mobile terminal 10 (the current location of the user) is an unvisited location.

[0115] Figure 12 It is a flowchart showing the processing sequence for determining the display mode of the AR object.

[0116] First, determine whether the current location is an unvisited location (step S21_1). That is, determine whether the current location exists in the location resume.

[0117] When the current location is an unvisited location, set the display mode of the AR object to the first display mode (step S21_2). The case of an unvisited location means that the current location does not exist in the location resume.

[0118] On the other hand, when the current location is not an unvisited location, that is, in the case of a visited location, set the display mode of the AR object to the second display mode (step S21_3). The case of a visited location means that the current location exists in the location resume.

[0119] In this way, determine the display mode of the AR object according to whether the current location is an unvisited location.

[0120] In the above series of processes, determine the type, arrangement area, and display mode of the AR object that appears in the live view image. Based on the determined matters, synthesize the AR object in the live view image to generate an AR image.

[0121] Figure 13 It is a conceptual diagram of AR image generation.

[0122] As Figure 13 shown, overlap the AR object ARO on a specified area of the captured image I (live view image) to generate an AR image ARI. In the present embodiment, the specified area refers to the area around the main subject MO and the ground.

[0123] The generated AR image is displayed on the display 15. That is, the image with the AR object overlapped is displayed on the display 15 (step S22). Thus, the AR image ARI can be viewed.

[0124] Then, determine whether the user has instructed to turn off the AR display (step S23). In step S13, when it is determined that the AR object is being displayed, also determine whether the user has instructed to turn off the AR display in the same way (step S23).

[0125] If an instruction to turn off the AR display is given, the process of the AR display is ended. The instruction to turn off the AR display is performed by a prescribed operation.

[0126] In step S12, when it is determined that the current position is not a specific area set in the map database 121, it is determined whether an AR object is being displayed (step S24). That is, it is determined whether an AR object is displayed in the live view image being displayed on the display 15. If it is determined that an AR object is being displayed, the display of the AR object is ended (step S25). That is, in this case, the user has moved from the specific area, so the display of the AR object is deleted. Then, it is determined whether the user has instructed to turn off the AR display (step S23), and if the AR display is instructed to be turned off, the process of the AR display is ended.

[0127] In step S16, when it is determined that there is no main subject in the captured image, it is similarly determined whether an AR object is being displayed (step S24). If it is determined that an AR object is being displayed, the display of the AR object is ended (step S25). That is, in this case, the main subject has disappeared from the captured image, so the display of the AR object is also deleted. Then, it is determined whether the user has instructed to turn off the AR display (step S23), and if the AR display is instructed to be turned off, the process of the AR display is ended.

[0128] In addition, the AR object displayed once can be configured to automatically disappear after a certain period of time.

[0129] As described above, according to the present embodiment, the type and display method of the AR object that appears are determined based on the information related to the environment of the space where the AR object is arranged and the information of the user's behavior history. Thereby, an AR space with excellent interestingness can be provided.

[0130] [Modification Example]

[0131] [Method for Determining the Type of AR Object]

[0132] In the above embodiment, it is configured to determine the type of the AR object that appears based on the recognition result of the object in the arrangement area of the AR object, but the method for determining the type of the AR object that appears is not limited to this. Based on various information that can be obtained in the mobile terminal 10, the type of the AR object that appears can be determined.

[0133] For example, it can be configured to determine the type of the AR object that appears based on the information of the current position of the mobile terminal 10 (the current position of the user). In this case, in the AR object database 123, the information of each AR object is recorded in association with the information of the area and the place.

[0134] Moreover, it is also possible to configure to determine the type of the AR object that appears based on the current position of the mobile terminal 10 (the current position of the user) and the information of the object recognized from the captured image. In this case, for example, it is possible to configure to determine the first classification of the AR object that appears based on the information of the current position of the mobile terminal 10, and determine the second classification of the AR object that appears based on the information of the object recognized from the captured image. The first classification is a so-called large classification, and the second classification is a classification obtained by subdividing the first classification (a so-called small classification).

[0135] Moreover, it is also possible to configure to perform scene recognition instead of or together with object recognition, and determine the type of the AR object that appears based on the recognized scene, or based on the recognized scene and object. In addition, since scene recognition itself is a well-known technology, the detailed description thereof is omitted. For example, a configuration can be adopted in which scene recognition of the captured image is performed using a learned model that has been machine-learned in a manner of performing scene recognition from an image.

[0136] In addition, when the mobile terminal 10 can acquire information such as the current date and time, temperature, humidity, sound, etc., these information can be acquired as environmental information, and the type of the AR object that appears can be determined based on the acquired information.

[0137] Moreover, the information of the user's behavior history can also be considered to determine the type of the AR object that appears. That is, the type of the AR object that appears can also be determined using both the environmental information and the information of the user's behavior history.

[0138] [Method for determining the display mode of the AR object]

[0139] In the above-described embodiment, as information on the user's behavior history, information on the location history is used to determine the display mode of the AR object. However, the information on the behavior history used to determine the display mode of the AR object is not limited thereto. Various information that can be obtained in the mobile terminal 10 can be used. For example, information on the usage history of this function (AR display function) in the mobile terminal 10, information on the operation history of the user in this function, etc. can be used. The usage history includes various usage-related information such as the total number of usage times, the total usage time, the number of usage times per day, the usage time per day, the usage time period, and the number of display times of each AR object. The information on the number of display times of each AR object is information on the number of appearances (how many times it has appeared) of each AR object. Also, the information on the operation history includes the shooting history of still images and moving images, information on the printing history, etc. The shooting history also includes information on the usage history of the camera unit 18. That is, it includes information on which of the internal camera unit 18A and the external camera unit 18B was used. The printing history is a history of printing the captured images (printing on tangible objects such as film, paper, media, etc.). There is no particular limitation on the printing method. Also, information on the usage history and operation history of the mobile terminal itself can be used. In the information on the usage history of the mobile terminal itself, for example, it includes information on the shooting history of using the mobile terminal 10. Also, the shooting history can include information on the analysis result of the captured images. In the analysis result of the captured images, information on the result of image recognition performed on the captured images (for example, the recognition result of the subject, the recognition result of the scene, etc.) can be included.

[0140] Also, the information on the behavior history used can be limited to information related to a specific event. For example, it can be limited to only information related to the current location. For example, regarding the number of shootings, it can be limited to the number of shootings in a specific area (the number of shootings in the current area). Also, regarding the number of printings, it can be limited to the number of printings in a specific area (for example, the number of printings in the current area) or the number of times a specific subject (for example, a subject related to the current area) has been printed.

[0141] Also, in the user's behavior history, it can further include the purchase history of goods, the posting history in a social networking service (SNS), etc. Regarding the purchase history of goods, for example, information on whether a specific good (for example, local specialties in the area, limited goods sold only in the area, etc.) has been purchased in the area, the number of purchase times, etc. can be used in association with the current location. Also, regarding the posting history in the SNS, information on whether a post has been made using information related to the current location, the number of posting times, etc. can be used.

[0142] By comprehensively using the information on the user's behavior history, it is possible to determine the display mode of the AR object that appears.

[0143] In addition, environmental information can also be considered to determine the display mode of the AR object that appears. That is, it is also possible to use both environmental information and information on the user's behavior history to determine the display mode of the AR object that appears.

[0144] Moreover, it can be configured such that when the behavior history satisfies certain conditions, a lottery, a game, etc. are implemented, and based on the result, the display mode of the AR object is determined.

[0145] Figure 14 FIG. is a flowchart showing an example of the processing sequence when determining the display mode of an AR object by implementing a lottery.

[0146] First, it is determined whether the current location is an unvisited location (step S21_11). When the current location is an unvisited location, the display mode of the AR object is set to the first display mode (step S21_2).

[0147] When the current location is not an unvisited location (the case of a visited location), it is determined whether it has been visited three or more times (step S21_13). That is, it is determined whether this visit is the third or more visit.

[0148] When it is not a visit of the third or more times (the case of the second visit), the display mode of the AR object is set to the second display mode (step S21_14).

[0149] On the other hand, when it is a visit of the third or more times, a lottery is implemented (step S21_15).

[0150] Figure 15 FIG. is a diagram showing an example of a lottery screen.

[0151] Figure 15 This is an example of a lottery using dice. When two dice both show 1 point when shaken, it is called "winning the lottery".

[0152] As Figure 15 shown, an image STA of a disk-shaped stage is superimposed on the live view image, and images DA1 and DA2 of two dice are displayed on the image STA of the stage. And a start button STB is displayed on the screen.

[0153] The user touches the start button STB to instruct the execution of the lottery. If the execution of the lottery is instructed, an animated image of the dice rotation is displayed. And, at the same time, the display of the start button STB is switched to the display of the stop button (not shown). The user touches the stop button to instruct the rotation of the dice to stop. If the rotation is stopped, it is switched to an image of the dice stopping and the number of points is determined. The number of points of each dice is randomly determined. For example, a random number is obtained within a numerical range of 1 to 6 to determine the number of points of each dice.

[0154] As described above, when the points of both dice are 1, it is determined as "win", and otherwise it is determined as "no win".

[0155] If a lottery is conducted, it is determined whether the result of the lottery is a winning one (step S21_16). If the result of the lottery is "winning", the display mode of the AR object is set to the third display mode (step S21_17). On the other hand, if the result of the lottery is "not winning", the display mode of the AR object is set to the second display mode (step S21_17).

[0156] Here, the number of AR objects displayed in the second display mode is set to be larger than that in the first display mode, and the number of AR objects displayed in the third display mode is set to be larger than that in the second display mode.

[0157] In this way, by introducing random elements when determining the display method, the interest can be further improved.

[0158] In addition, in addition to complete randomization, randomization can also be performed with a deviation. For example, if certain conditions are met, randomness can be controlled to obtain favorable results. For example, the results can be different based on the user's usage status (for example, whether there is a charge), the number of past appearances, the number of shots, the number of prints, etc. of the AR object that appeared.

[0159] [How AR objects are displayed]

[0160] In the above-described embodiment, it is configured to change the number of AR objects that appear according to the user's behavior history, but the elements to be changed are not limited to this. It can be configured to change the so-called gorgeousness, luxury, etc. That is, it can be configured to change the level of appearance. For example, in the case of changing the display mode of the AR object according to the position history, it can be set that the higher the access count, the higher the level of appearance (for example, a more gorgeous display, a more luxurious display, etc.). And, for example, in the case of changing the display mode of the AR object according to the number of still image captures, it can be set that the higher the number of captures, the higher the level of appearance. And, for example, in the case of changing the display mode of the AR object according to the display count (appearance count), it can be set that the higher the display count, the higher the level of appearance.

[0161] And, in the above-described embodiment, it is configured to switch between two display modes, but it is also possible to further prepare multiple display modes and switch the display mode in more detail according to the user's behavior history.

[0162] [Change in Display Mode Based on Specific Behavior]

[0163] When the user performs a specific behavior, the display mode of the AR object can be further changed. For example, when the user captures an image showing the AR object, the display mode of the AR object can be further changed. Capturing includes both still image capture and moving image capture. And, when the captured image is printed, the display mode of the AR object can be further changed. In this case, as the change method, it is possible to adopt further increasing the number of AR objects, further increasing the level of appearance, etc. The still image and moving image capture instructions and the printed image capture instructions are examples of the output instructions of the second image. In addition, capturing includes directly obtaining the content displayed on the screen of the display 15 as image data (so-called screenshot, screen capture).

[0164] [Configuration Area of AR Object]

[0165] In the above-described embodiment, the main subject (person) is recognized and the area around it and on the ground is set as the configuration area of the AR object, but the method for determining the configuration area of the AR object is not limited to this. It is preferable to set the configuration area according to the type of the AR object that appears, etc. For example, for an unnatural AR object floating in the air, it is not necessarily required to be configured on the ground. It can also be configured in the air, on the water surface, etc. according to the type of the AR object. Similarly, for an AR object for demonstration (colored paper pieces, twinkling of stars, etc.), it is not necessarily required to be configured on the ground.

[0166] Further, it is not necessarily required to be arranged around the main subject, but can be configured to be arranged at a specific location within the screen. For example, it can be configured to be arranged at the center of the screen. In this case, it is not necessary to identify the main subject.

[0167] Moreover, when the mobile terminal 10 has a spatial recognition function, it is preferable to use its recognition result to determine the arrangement area. For example, when the mobile terminal 10 has a ranging sensor such as LiDAR, its ranging result can be used to determine the arrangement area.

[0168] [Second Embodiment]

[0169] In the present embodiment, a case where a character created by CG (a person, animal, doll, robot, etc. appearing in a novel, movie, drama, comic, computer game, etc.) is overlapped on a captured image to generate an AR image will be described.

[0170] In the present embodiment, the type and attributes of the character appearing in the image are determined based on the environmental information and the behavior history information. More specifically, the type of the character is determined based on the environmental information, and the attributes of the character are determined based on the behavior history information. In addition to the appearance elements (so-called shape) such as the character's appearance, clothing, decoration, equipment, growth level, evolution level, etc., the attributes also include elements other than the appearance such as intimacy, personality, and status. For example, in the case of a character having a conversation, the conversation content, actions, etc. change according to the intimacy, personality, etc.

[0171] In the present embodiment, the case of performing AR display using a mobile terminal is also taken as an example for description. The basic structure of the mobile terminal is the same as that of the first embodiment described above. Therefore, only the differences, that is, the AR object (character) that appears and its determination method, will be described here.

[0172] In addition, in the present embodiment, the case of changing the appearance elements of the character is taken as an example for the attributes of the character.

[0173] [AR Object and Its Determination Method]

[0174] As described above, in the present embodiment, an AR image in which a character appears is generated. The character (AR object) that appears is composed of 3DCG, for example, and its data is recorded in the AR object database 123. The selection of the character (AR object) is performed by the AR object determination unit 114.

[0175] Figure 16 It is a block diagram of the main functions of the AR object determination unit.

[0176] The AR object determination unit 114 has functions such as a main subject determination unit 114A, a placement area determination unit 114B, a first element determination unit 114C, and a second element determination unit 114D.

[0177] The functions of the main subject determination unit 114A and the placement area determination unit 114B are the same as those in the first embodiment described above. The main subject determination unit 114A determines the main subject based on the recognition result (object information) of the object recognition unit 111B. The placement area determination unit 114B determines the placement area of the character based on the determination result of the main subject by the main subject determination unit 114A.

[0178] The first element determination unit 114C determines the type of the character as the first element based on the information of the current position of the mobile terminal 10 (current position information). The current position of the mobile terminal 10 is determined by the position determination unit 111A. The first element determination unit 114C refers to the information recorded in the AR object database 123 to determine the type of the character.

[0179] Figure 17 It is a diagram showing an example of the information recorded in the AR object database.

[0180] As Figure 17 shown, information on the characters that appear for each specific point of interest on the map is recorded in the AR object database 123. The points of interest on the map use the points of interest registered in the map database 121.

[0181] A character ID (Identification: identification information) is assigned to each character as identification information, and appearance information (data of 3DCG) is recorded in association with the character ID.

[0182] The first element determination unit 114C selects the character corresponding to the current position of the mobile terminal 10 (the current position of the user) from among the multiple characters recorded in the AR object database 123. For example, according to Figure 17 the example, when the current position is the point of interest A, the character with the character ID: 1000000A is selected.

[0183] The second element determination unit 114D determines the appearance of the character based on the information related to the user's behavior history acquired by the behavior history information acquisition unit 112. Similar to the first embodiment, the second element determination unit 114D determines the appearance of the character based on the user's location history information. Therefore, the appearance is determined according to whether the place is the first visit of the user. In the case of the place where the user visits for the first time (the case of an unvisited location), the second element determination unit 114D sets the appearance of the character to the first appearance. On the other hand, in the case of a place that is not the first visit of the user (the case of a visited location), the second element determination unit 114D sets the appearance of the character to the second appearance.

[0184] Figure 18 It is a diagram showing an example of the appearance of the character.

[0185] As Figure 18 shown, different appearances are presented in the first appearance and the second appearance. The second appearance is set to a higher level of appearance than the first appearance. That is, the appearance becomes more magnificent and luxurious. In Figure 18 the example of, it is an example where the clothing (clothes), decoration, and equipment are changed. In addition, it can also be set as a setting where the decoration, equipment, growth level, evolution level, etc. are changed.

[0186] [Actions of the mobile terminal]

[0187] The processing sequence during AR display is substantially the same as that of the first embodiment. Therefore, here, with reference to Figure 11 , the processing sequence during AR display will be described.

[0188] First, a live view image is displayed on the display 15 (step S10). Next, information on the current position is acquired (step S11). The mobile terminal 10 acquires information on the current position (the user's current position) of the mobile terminal 10 based on the GPS signal received by the GPS receiving unit 17. Next, based on the acquired information on the current position, it is determined whether the current position is a specific area set in the map database 121 (step S12). When the current position is the specific area determined in the map database 121, it is determined whether an AR object is displayed in the live view image being displayed on the display 15 (step S13). When the AR object is not displayed, a process of recognizing an object from the captured image is performed (step S14). Based on the recognition result of the object, the main subject is determined from the captured image (step S15). In the present embodiment, a person is also set as the main subject. Whether there is a main subject is determined based on the result of the determination process of the main subject (step S16). That is, it is determined whether there is a main subject in the captured image. When there is a main subject in the captured image, an arrangement area for a character (AR object) is determined based on the position of the main subject in the image (step S17). In the present embodiment, an area on the ground around the main subject is also set as the arrangement area for the character.

[0189] After determining the arrangement area for the AR object, environmental information is acquired (step S18). In the present embodiment, information on the current position (the user's current position) of the mobile terminal 10 is acquired as the environmental information. The current position of the mobile terminal 10 is determined by the position determination unit 111A based on the GPS signal received by the GPS receiving unit 17.

[0190] Based on the acquired current position information, the type of the character (AR object) that appears is selected (step S19). The type of the character that appears is selected from the AR object database 123.

[0191] If the character that appears is determined, information on the user's behavior history is acquired (step S20). In the present embodiment, information on the position history is also acquired as the information on the behavior history. Based on the acquired information on the position history, the display mode of the character (AR object) is determined (step S21). In the present embodiment, the display mode of the character that appears is determined based on whether the current position is an unvisited location.

[0192] Figure 19 It is a flowchart showing the order of the determination process of the appearance of the character.

[0193] First, it is determined whether the current position is an unvisited location (step S21_11). That is, it is determined whether the current position exists in the position history.

[0194] When the current location is an unvisited location, set the appearance of the appearing character to the first appearance (step S21_12).

[0195] On the other hand, when the current location is not an unvisited location, that is, when it is a visited location, set the appearance of the appearing character to the second appearance (step S21_13).

[0196] In this way, the appearance of the appearing character is determined according to whether the current location is an unvisited location.

[0197] In the above series of processes, the type, arrangement area, and appearance of the character (AR object) appearing in the image are determined. According to the determined matters, the character is synthesized in the live view image to generate an AR image. That is, the AR image is generated by overlapping and displaying the character in a specified area of the captured image (live view image). In the present embodiment, the specified area refers to the area around the main subject and the ground.

[0198] The generated AR image is displayed on the display 15. That is, the image with the character (AR object) appearing is displayed on the display 15 (step S22). Thus, the AR image can be viewed.

[0199] Figure 20 It is a diagram showing an example of AR display.

[0200] As Figure 20 shown, the character (AR object) ARO composed of 3DCG is overlapped and displayed in a specified area of the captured image I (live view image).

[0201] As described above, according to the present embodiment, when generating an AR image of a character, the type and attributes of the appearing character are determined according to the information of the environment of the space where the character is arranged and the information of the user's behavior history. Thus, an AR space with excellent interestingness can be provided.

[0202] [Modification Example]

[0203] [Method for Determining Character Type]

[0204] In the above embodiment, as the environmental information, the information of the current location of the mobile terminal 10 (the current location of the user) is used to determine the type of the appearing character (AR object), but other information can also be used to determine the type of the character. For example, similarly to the above first embodiment, the type of the character can be determined according to the information of the recognition result of the object in the arrangement area of the character. And, the shooting scene can be recognized, and the type of the appearing character can be determined according to the recognized scene. In addition, the information of the behavior history can also be considered to determine the type of the appearing character.

[0205] [How to determine the character's appearance]

[0206] In the above-mentioned embodiment, the appearance of the character is determined by using the information of the location history as the information of the user's behavior history, but the appearance of the character can also be determined by using other information. Various information that can be obtained in the mobile terminal 10 can be used. Therefore, the result of environmental recognition (environmental information) can also be used. This is the same as the point that the display method of the AR object can be determined by using various information. Therefore, it can be configured to implement a draw, a game, etc. when the behavior history meets certain conditions, and determine the appearance of the character based on the result.

[0207] [Changes in appearance based on specific behavior]

[0208] When the user performs a specific behavior, the appearance of the character can be further changed. For example, when the user takes a picture (including the shooting of a dynamic image) showing an image of the character, the appearance of the character can be further changed. And when the photographed image is printed, the appearance of the character can be further changed. In this case, the randomly selected appearance can be changed. Thus, the fun can be further improved.

[0209] Furthermore, when the user performs a specific behavior, the character can be caused to perform a specific action. For example, when the user takes a picture (including a dynamic picture) showing an image of a character, the character can be caused to perform a specific action. Furthermore, when the taken image is printed, the character can be caused to perform a specific action.

[0210] Figure 21 This figure shows an example of making a character perform a specific action by shooting.

[0211] Figure 21 (A) indicates the display before shooting. Figure 21 (B) indicates the display after shooting. Figure 21 As shown, by shooting a still image, the character (AR object) ARO performs an action expressing joy. In this way, as a specific action, for example, an action expressing joy can be exemplified. In addition, an action to prompt printing (an action to request printing) can also be performed. In addition, a randomly selected action can also be performed. In this way, the fun can be further improved.

[0212] This action continues for a specified time, and then returns to normal display.

[0213] Figure 22 This is a diagram showing an example of display during printing.

[0214] Figure 22This represents an example of printing by a so-called mobile printer. In particular, an example is shown when using a printer 100 (a so-called instant printer) that prints an image on an instant film 110. The printer 100 and the mobile terminal 10 perform wireless communication via Wifi (registered trademark), Bluetooth (registered trademark), etc., and transmit and receive data. In addition, since such a printer 100 and the printing method itself are well-known, a description of its detailed content is omitted.

[0215] As Figure 22 shown, when printing is performed, the image of the printed character is displayed on the display 15 and a prescribed action is performed. This action continues for a prescribed time. For example, until the transmission of printing data to the printer is completed.

[0216] [Character attributes]

[0217] In the above-described embodiment, the appearance of the character is changed according to the user's behavior history, but the elements to be changed are not limited thereto. As described above, the attributes include not only appearance elements such as the character's facial features, clothing, decorations, equipment, growth level, evolution level, etc., but also elements other than appearance such as intimacy, personality, status, etc. For example, it can be configured to increase intimacy according to the behavior history.

[0218] Regarding the attributes of the character, it is preferable to record the history of changes. Thereby, the attributes can change step by step. For example, in the case where the attributes change according to the location history, even when accessing the same place where the first attribute is the first attribute, the second attribute is the second attribute, and the third attribute is the third attribute during the second visit, etc., the attributes can change step by step according to the number of visits.

[0219] Figure 23 This is a diagram showing an example of history data of changes in the attributes of a character.

[0220] As Figure 23 shown, each character is managed by a character ID, and the number of appearances, the number of shootings, the number of printings, the category, the number of appearance changes, the intimacy, etc. are recorded separately. The number of appearances is the number of times the character appears (the number of occurrences). The number of shootings is the number of times the AR image of the appearance of the character is shot. The number of printings is the number of times the AR image of the appearance of the character is printed. The category is the category (type) of the character. The number of appearance changes is the number of times the appearance of the character is changed. The intimacy is the intimacy of the character. The intimacy changes, for example, according to the number of appearances, the appearance frequency (the number of appearances within a prescribed period), the period after the last appearance, etc.

[0221] In addition, the attributes of the character change according to the user's behavior history. Therefore, the history of the change in the character's attributes is also the user's behavior history. Therefore, by using the information of the history of the change in the character's attributes, the attributes of the character can also be changed. The information of the history of the change in the character's attributes is an example of the information of the history related to the objects that appear. The information of the history of the change in the character's attributes is recorded in the user database as user data, for example. Or, it is recorded in the behavior history database.

[0222] The attributes of the character change according to the relationship or interaction with the user. Therefore, the affection for the character is further enhanced, and the fun can be further improved. That is, the more the number of appearances, the number of shootings, or the number of printings increases, the higher the level of the appearance, or the character grows or evolves, or the intimacy increases, so it is easier to generate a sense of attachment.

[0223] [Character placement area]

[0224] In the above embodiment, the area around the main subject and on the ground is set as the character placement area, but the method of determining the character placement area is not limited to this. As described in the modification example of the first embodiment above, it is preferable to set the placement area according to the type of the character, etc. In addition, the character placement area can also be determined according to the information of the user's behavior history. For example, when determining the character placement area according to the user's location history information, the placement area is determined as follows.

[0225] Figure 24 It is a conceptual diagram showing an example of the placement of the character.

[0226] Figure 24 (A) shows an example of the AR image generated when the current location is an unvisited location. And, Figure 24 (B) shows an example of the AR image generated when the current visited location is a visited location.

[0227] As Figure 24 shown in (A), when the current location is a visited location, the character (AR object) ARO is placed in the area at a distance of D1 or more and less than D2 from the main subject MO (distance D1 < distance D2).

[0228] On the other hand, when the current location is an unvisited location, as Figure 24 shown in (B), the character ARO is placed in the area at a distance of D2 or more and less than D3 from the main subject MO (distance D2 < distance D3).

[0229] In this way, when the current location is a visited location, the character ARO is placed closer to the main subject MO than in the case of an unvisited location.

[0230] In determining the character's placement area, in addition to this, information about the number of appearances, number of shots, number of prints, and intimacy of the character can also be used. In this case, for example, the more the number of appearances, the more the number of shots, the more the number of prints, and the higher the intimacy, the closer the character is to the main subject. This information can be used in combination, or each item can be weighted for determination and the placement position can be determined.

[0231] Furthermore, image recognition (including object recognition) can be performed on the image in which the character appears, and the result can be used to determine the configuration area and configuration position of the character. For example, when a specific subject (e.g., a sign, a monument, etc.) is recognized from the image, the configuration position of the character can be determined by configuring the character side by side with the specific subject. Furthermore, for example, when a specific subject is recognized from the image and a person is recognized, the configuration position of the character can be determined by configuring the specific subject between the character and the person.

[0232] Figure 25 This is a conceptual diagram showing an example of the arrangement of characters using image recognition.

[0233] Figure 25 This shows an example of taking a selfie with a specific subject (here, a monument M) included in the screen. Selfie refers to a method of taking a photo with the photographer himself as the subject (selfie). In this case, the position of the character ARO is determined by placing the monument M between the photographer (user) MO as the main subject and the character (AR object) ARO.

[0234] The recognition of the specific subject can be performed by using a known image recognition technology. Also, since self-portraits are usually taken using the internal camera unit 18A, the determination of whether it is a self-portrait can be made based on whether the image is taken by the internal camera unit 18A.

[0235] [Other embodiments]

[0236] [Random Print]

[0237] When printing an image showing an AR object such as a character, the display of the AR object can be changed and printed. And the change can be made randomly. For example, when printing an image showing a character, the character's expression, posture, clothing, etc. can be changed randomly and printed. This can further increase the fun of printing.

[0238] Figure 26 It is a concept map for printing.

[0239] Figure 26 An example in which an image is printed on the instant film 110 is shown.Figure 26 (A) indicates an image for which printing is instructed. Figure 26 In the example, the image displayed on the display 15 of the mobile terminal 10 is an image for which printing is instructed. Figure 26 (B) shows the result of printing. The image instructed to be printed is printed with the character's expression, posture, and decoration changed.

[0240] Unlike just displaying on a display, printing has the fun of obtaining a unique image that cannot be modified. Therefore, when printing, by changing the display of the AR object, the user can obtain an unpredictable unique image, which can further increase the fun. In addition, by making the change random, the fun can be further increased. The random element can be determined, for example, based on the time when printing is instructed.

[0241] [Expansion towards cross reality (XR)]

[0242] In the above embodiments, the present invention is applied to processing in augmented reality (AR) as an example, but the application of the present invention is not limited thereto and can also be applied to processing in mixed reality (MR), virtual reality (VR), etc.

[0243] MR is a technology that further develops AR. It is a technology that can synthesize virtual objects (objects) with spatial expansibility into specific positions in the real space in real time, and operate or intervene through physical intervention such as touching with hands. MR has three-dimensional spatial recognition capabilities and can integrate a virtual world with spatial information into the real world. In MR, the device uses multiple cameras, sensors, etc. to recognize spatial information and grasp the shape, position information, etc. of the real world, so that a virtual world with spatial information can be expressed in the real world. MR is usually provided using a dedicated head-mounted display (Head Mounted Display; HMD). In MR, the camera and various sensors possessed by the device (for example, HMD) can also be used to obtain information on the environment of the space where the character is configured.

[0244] In the case of VR, everything is provided in CG. Therefore, in this case, the mobile terminal is also displayed in CG as a virtual object in the virtual space. The user operates the mobile terminal as a virtual object through the avatar as the avatar in the virtual space. In the virtual space, the user's position is recognized as the position of the avatar. The space where the object appears is recognized as the space captured by the mobile terminal used by the avatar in the virtual space.

[0245] In recent years, Digital Twin, which reproduces the real space in a virtual space, has gradually become a reality. Therefore, the present invention can also be implemented even in a virtual space that reproduces the real space.

[0246] In addition, in the case of VR, the space constituting the field-of-view area of the avatar can be set as the configuration space of the object, and the object that appears in the virtual space can be determined based on the information related to the environment of this space and the information related to the user's behavior history. In this case, the information related to the environment is obtained from the image of the field-of-view area constituting the avatar. For example, the information of the objects in the image of the field-of-view area constituting the avatar is obtained. Also, the information of the position (coordinate position) of the avatar in the virtual space is obtained to obtain the information of the user's position.

[0247] Thus, the present invention is not limited to the real space and can also be applied to the case where an object appears in a virtual space (such as the metaverse, etc.).

[0248] [System Structure]

[0249] In the above-described embodiment, it is configured that the mobile terminal performs all the processing, but it can also be configured that a part of the processing is performed by other devices. For example, it can be configured that a part of the processing is performed by the server. For example, it can be configured that the mobile terminal only performs shooting, various sensing, and display of AR images, and the management of various databases, generation of AR images, etc. are performed by the server. In this case, the communication method between the mobile terminal and the server is not particularly limited.

[0250] Also, in the above-described embodiment, it is configured that image shooting, various sensing, and image display are all performed by the mobile terminal, but it can also be configured that each processing is performed by different devices. For example, it can be configured that the image display is performed using an HMD.

[0251] [Hardware Structure]

[0252] The functions implemented by the information processing apparatus are realized by various processors. Among the various processors, there are a general-purpose processor, i.e., a CPU and / or a GPU (Graphic Processing Unit), which executes a program and functions as various processing units, a programmable logic device (PLD) such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit structure can be changed after manufacturing, and an ASIC (Application Specific Integrated Circuit), which is a dedicated circuit, i.e., a processor having a circuit structure specifically designed to execute a specific process. A program has the same meaning as software.

[0253] One processing unit may be constituted by one of these various processors, or may be constituted by two or more processors of the same type or different types. For example, one processing unit may be constituted by multiple FPGAs, or a combination of a CPU and an FPGA. Further, multiple processing units may be constituted by one processor. As an example of constituting multiple processing units by one processor, first, there is the following method: represented by a computer for a client and a server, etc., one processor is constituted by a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, there is the following method: represented by a system on chip (SoC), etc., a processor that uses one IC (Integrated Circuit) chip to realize the functions of the entire system including multiple processing units is used. Thus, the various processing units are constituted by using one or more of the above various processors as a hardware structure.

[0254] Symbol Explanation

[0255] 10 - Mobile terminal, 11 - CPU, 14 - Built-in memory, 15 - Display, 16 - Touchpad, 17 - GPS receiver, 18 - Camera unit, 18A - Internal camera unit, 18B - External camera unit, 19 - Microphone unit, 20 - Speaker unit, 21 - Wireless communication unit, 22 - Near-field wireless communication unit, 23 - Sensor unit, 30 - Touch panel, 100 - Printer, 110 - Instant film, 111 - Environmental information acquisition unit, 111A - Location determination unit, 111B - Object recognition unit, 112 - Behavior history information acquisition unit, 113 - Behavior history database management unit, 114 - AR object determination unit, 114A - Main subject determination unit, 114B - Configuration area determination unit, 114C - First element determination unit, 114D - Second element determination unit, 115 - AR image generation unit, 116 - Display control unit, 117 - Recording control unit, 121 - Map database, 122 - Behavior history database, 123 - AR object database, ARI - AR image, ARO - AR object, DA1 - Image of dice, DA2 - Image of dice, I - Captured image, M - Memorial, MO - Main subject, R - Configuration area, SSB - Shutter button, STA - Image of stage, STB - Start button, S10~S25 - Action sequence of the mobile terminal during AR display, S21_1~S21_3 - Determination processing sequence of the display mode of the AR object, S21_11~S21_17 - Processing sequence when performing a lottery to determine the display mode of the AR object.

Claims

1. An information processing apparatus, comprising a processor, wherein the processor performs the following processing: acquiring first information related to the environment of the space where an object appears; acquiring second information related to the user's behavior history; determining a first element related to the object based on the first information; determining a second element related to the object based on the second information; and determining the object that appears in the space based on the first element and the second element.

2. The information processing apparatus according to claim 1, wherein the processor performs the following processing: acquiring a first image that captures the space where the object appears; and generating a second image with the object superimposed on the first image.

3. The information processing apparatus according to claim 2, wherein the processor processes the first image to identify an object within the first image or a scene of the first image, and acquires the first information.

4. The information processing apparatus according to claim 3, wherein the processor performs the following processing: acquiring information on the position of the user; and when the user is present in a specific area, identifying an object within the first image or a scene of the first image, and acquiring the first information.

5. The information processing apparatus according to claim 1, wherein the second information includes information on the user's movement history or information on the history related to the object that appears.

6. The information processing apparatus according to any one of claims 1 to 5, wherein the first element is the type of the object that appears, the second element is the display mode or attribute of the object that appears.

7. The information processing apparatus according to claim 6, wherein the processor performs the following processing: processing an image that captures the space where the object appears to identify a main subject; and determining a configuration area of the object that appears in the space based on the position of the main subject within the space.

8. The information processing apparatus according to claim 7, wherein the processor determines the type of the object that appears based on the first information within the configuration area.

9. The information processing apparatus according to claim 6, wherein the processor randomly determines the display mode of the object when the user's behavior history meets specific conditions.

10. The information processing apparatus according to any one of claims 2 to 4, wherein the processor changes the display mode or attribute of the object according to an output instruction of the second image.

11. An information processing method, comprising the following steps: a step of acquiring first information related to the environment of the space where an object appears; a step of acquiring second information related to the user's behavior history; a step of determining a first element related to the object based on the first information; a step of determining a second element related to the object based on the second information; and a step of determining the object that appears in the space based on the first element and the second element.

12. An information processing program that causes a computer to implement the following functions: A function of obtaining first information related to the environment of the space where an object appears; A function of obtaining second information related to the user's behavior history; A function of determining a first element related to the object based on the first information; A function of determining a second element related to the object based on the second information; and A function of determining the object that appears in the space based on the first element and the second element.

13. A recording medium that is non-volatile and readable by a computer, and records the program according to claim 12.

Images