Image processing apparatus and image processing method

Abstract

One embodiment of the technology of the present invention provides an image processing apparatus and an image processing method for processing an image in a virtual space. An image processing device according to one embodiment of the present invention is provided with: a processor; and a memory that stores a program to be executed by the processor, the processor performing, with reference to the memory, processing of: acquiring information of a virtual space including three-dimensional position information; acquiring a specific position in the virtual space; acquiring evaluation information related to evaluation of the specific position; determining the evaluation information on the basis of a preset reference; and changing the image at the specific position in the virtual space using a first image, which is an image of the real world corresponding to the specific position, on the basis of the result of the determination.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing apparatus and an image processing method for processing an image in a virtual space. BACKGROUND

[0002] As for a technology of processing an image related to a virtual space, for example, a technology of providing a virtual content corresponding to a prescribed place in a real world is described in Patent Literature 1.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2022-122810 SUMMARY

[0006] One embodiment of the technology of the present application provides an image processing apparatus and an image processing method for processing an image in a virtual space.

[0007] Means for solving technical problems

[0008] The image processing apparatus according to the first aspect of the present application includes a processor and a memory storing a program for causing the processor to execute the following processing with reference to the memory: acquiring information of a virtual space including three-dimensional position information; acquiring a specific position in the virtual space; acquiring evaluation information related to evaluation of the specific position; judging the evaluation information based on a predetermined criterion; and changing an image at the specific position in the virtual space using a first image that is an image of a real world corresponding to the specific position, according to a result of the judgment.

[0009] According to the first aspect, the image at the specific position in the virtual space is changed using the first image that is an image of a real world corresponding to the specific position, according to a result of the judgment of the evaluation information. The processor can cause a user of the image processing apparatus to view the changed image.

[0010] In the first aspect and each of the following aspects, the "specific position" is, for example, a place viewed by a user of the image processing apparatus at a visiting place of the virtual space, but can be another place focused on by the user. The processor can change the image at the specific position when it is judged that the evaluation information at the specific position satisfies the criterion. As for the "evaluation information", the criterion or scale of the evaluation can be one or a plurality of. Further, the "information of the virtual space" includes three-dimensional position information, and can have information of a shape, a color, a brightness, a pattern, and the like in addition to this. Some images can be displayed or mapped in an initial state (before the image is changed).

[0011] Further, the image processing apparatus according to the first aspect can be configured as a server apparatus on a network, but is not limited to this.

[0012] The image processing apparatus according to the second aspect, in the first aspect, changes the image at the specific position in the virtual space using the first image when the evaluation value calculated based on the evaluation information is equal to or higher than the threshold value.

[0013] The image processing apparatus according to the third aspect, in the first aspect or the second aspect, judges at least one of the following as a reference: the number of photographs of the first image for the specific position; the photographer of the first image for the specific position; the number of positive evaluations for the specific position; the comment of the user of the image processing apparatus for the specific position; and the biometric information of the photographer associated with the first image for the specific position.

[0014] The image processing apparatus according to the fourth aspect, in any one of the first aspect to the third aspect, selects the first image based on a condition specified by the user of the image processing apparatus or a condition determined by the processor.

[0015] The image processing apparatus according to the fifth aspect, in any one of the first aspect to the fourth aspect, determines the photographing place of the image of the real world included in the first image group, and selects the first image based on the photographing place. Further, the "photographing place" refers to, for example, the position where the photographer or the photographic device exists when the image of the real world is photographed, and includes at least latitude and longitude, but can include altitude as well. Also, the photographing place can be expressed by a three-dimensional orthogonal coordinate system, instead of (latitude, longitude, altitude).

[0016] The image processing apparatus according to the sixth aspect, in the fifth aspect, determines the photographing place using at least one of feature amount extraction using a method of machine learning and pattern matching between images.

[0017] The image processing apparatus according to the seventh aspect, in the sixth aspect, determines at least one of the azimuth of the photographing, the elevation angle of the photographing, and the magnification of the photographing in addition to the photographing place, and selects the first image from the first image group based on at least one of the azimuth of the photographing, the elevation angle of the photographing, and the magnification of the photographing and the photographing place.

[0018] The image processing apparatus according to the eighth aspect, in any one of the first aspect to the seventh aspect, in the changing, changes the image of the virtual space corresponding to the photographing range of the first image and the image of the virtual space corresponding to the area outside the photographing range continuously in space or in time.

[0019] The image processing apparatus according to the ninth aspect, in any of the first through eighth aspects, wherein the processor changes in accordance with an operation or a motion predetermined by a user of the image processing apparatus.

[0020] The image processing apparatus according to the tenth aspect, in any of the first through ninth aspects, wherein the processor adjusts a degree of change in accordance with an attribute of a user of the image processing apparatus.

[0021] The image processing apparatus according to the eleventh aspect, in the tenth aspect, wherein the processor adjusts the degree of change using one or more of an age, a gender, and a preference of the user as the attribute.

[0022] The image processing apparatus according to the twelfth aspect, in any of the first through eleventh aspects, wherein the processor changes the first image using an image uploaded to the image processing apparatus by a user of the image processing apparatus as the first image.

[0023] The image processing apparatus according to the thirteenth aspect, in any of the first through twelfth aspects, wherein the processor accesses a website available to the public via a network and uses an image acquired from the website as the first image for the change.

[0024] The image processing apparatus according to the fourteenth aspect, in any of the first through thirteenth aspects, wherein the processor sets a sound corresponding to a specific location based on a sound associated with the first image when changing the first image.

[0025] The image processing apparatus according to the fifteenth aspect, in any of the first through fourteenth aspects, wherein the processor presents information related to the specific location to a user of the image processing apparatus.

[0026] The image processing apparatus according to the sixteenth aspect, in the fifteenth aspect, wherein the processor presents at least one of the following information to the user as the information related to the specific location: an explanation of a place in a real world corresponding to the specific location, that is, a visit candidate; a website introducing the visit candidate; a visit address of a travel agency handling a travel product for the visit candidate; and a visit address of a sales shop selling a souvenir of the visit candidate.

[0027] The image processing apparatus according to the seventeenth aspect, in any of the first through sixteenth aspects, wherein the processor changes the first image using an image subjected to image processing as the first image.

[0028] The image processing apparatus according to the eighteenth aspect, in the seventeenth aspect, wherein the processor performs the image processing on the first image based on an instruction of a user of the image processing apparatus.

[0029] The image processing apparatus according to the 19th aspect further includes an image storage device that stores the first image, and the processor performs the change based on an image selected from the first image stored in the image storage device.

[0030] The image processing apparatus according to the 20th aspect, in any one of the first aspect to the 19th aspect, the processor determines the specific position based on an output of a sensor that is worn by a user of the image processing apparatus and that detects a position and a posture of the user.

[0031] The image processing apparatus according to the 21st aspect, in any one of the first aspect to the 20th aspect, the processor displays the virtual space on a goggle-type device that is worn by a user of the image processing apparatus and that includes a display device.

[0032] The image processing method according to the 22nd aspect of the present application is executed by an image processing apparatus including a processor and a memory that stores a program for causing the processor to execute, in the image processing method, the processor performs the following processing with reference to the memory: acquiring information of a virtual space including three-dimensional position information; acquiring a specific position in the virtual space; acquiring evaluation information related to evaluation of the specific position; judging the evaluation information based on a predetermined criterion; and changing an image at the specific position in the virtual space using a first image that is an image of a real world corresponding to the specific position, according to a result of the judgment. The image processing method according to the 22nd aspect can have the same structure as the second aspect to the 21st aspect. Also, an image processing program that causes a computer to execute the image processing method of these aspects, and a non-transitory and tangible recording medium that records a computer-readable code of such an image processing program can also be cited as aspects of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a conceptual diagram illustrating a scenario of changing an image in a virtual space.

[0034] Figure 2 is a conceptual diagram illustrating a scenario of changing a part of an image of an object.

[0035] Figure 3 is a diagram illustrating a structure of an image processing system according to the first embodiment.

[0036] Figure 4 is a diagram illustrating a structure of an image processing server.

[0037] Figure 5 is a diagram illustrating information recorded in a database.

[0038] Figure 6is a diagram showing a scene of displaying an image and evaluation information.

[0039] Figure 7 is a diagram showing a structure of a user system in the first embodiment.

[0040] Figure 8 is a diagram showing a structure of a goggle.

[0041] Figure 9 is a flowchart (1 / 2) showing processing of an image processing method in the first embodiment.

[0042] Figure 10 is a flowchart (2 / 2) showing processing of an image processing method in the first embodiment.

[0043] Figure 11 is a diagram showing an example of information displayed on a display.

[0044] Figure 12 is a diagram showing a scene of changing an image.

[0045] Figure 13 is a diagram showing an example of display of information related to an access site or the like.

[0046] Figure 14 is a flowchart showing processing of acquisition and recording of an image.

[0047] Figure 15 is a diagram showing an example of changed image processing corresponding to a photographing time. DETAILED DESCRIPTION

[0048] [Sharing experience using virtual space content]

[0049] In recent years, due to the spread of the novel coronavirus infection, movement has been restricted, and travel has not been possible at will. In this case, as a solution to the sharp decline in tourists worldwide, travel experiences in a virtual reality space (Virtual Reality: VR, hereinafter, sometimes referred to as “VR space”, “virtual reality space”, or simply “virtual space”) are becoming popular. However, the content of the VR space itself is produced by a platform provider, and the experiential person receives a travel experience service within the space, so the experience content is exactly the same regardless of the user. That is, the content of the VR space does not reflect the experience when the user actually visits, and therefore the appeal and value of the VR experience are insufficient.

[0050] The VR-based sightseeing content is provided before / while / after the trip, and as the use purpose of the content after the trip, "experiencing the visited spot after the visit to stimulate the desire to visit again" and "recommending (sharing the excitement) to others" can be considered (see Non-Patent Literature 1), and thus if the experience at the actual visit can be reflected in the content of the VR space, it will help to further enhance the VR experience value (stimulate the desire to visit again, share the excitement).

[0051] Non-Patent Literature 1: "VR Utilization for Tourism Resource Heightening", Prime Minister's Official Residence Tourism Strategy Implementation Promotion Working Group (18th), 2018, [Retrieved on March 6, 2023], Internet (https: / / www.kantei.go.jp / jp / singi / kanko_vision / kankotf_dai18 / siryou5.pdf)

[0052] In view of such circumstances, the present inventors have conducted intensive studies, and have obtained the present application. Hereinafter, the specific modes of the present application (image processing apparatus and image processing method) will be described with reference to the accompanying drawings.

[0053] [1st Embodiment]

[0054] [Change of image at specific position in virtual space]

[0055] In the 1st embodiment of the present application, in a VR space browsable in a state where a virtual reality (VR) display device is worn, when the VR space is a representation that imitates a real world, a key experience area (key experience space) in the VR space is determined based on a still image / video image (1st image) captured in the real world at an original position or based on an image (1st image) based on these images, and the color, brightness, contrast, etc. of the area and space in the VR space are converted based on these data (images).

[0056] [Change of image in virtual space]

[0057] Figure 1 is a conceptual diagram showing a scenario of image change in the present application. As shown in part (a) of Figure 1 As shown in part (a) of

[0058] Figure 1 Part (b) shows an image (image) corresponding to a specific location in the virtual space (first image). In the first embodiment, as described in detail below, the image processing server 20 (processor 22) can perform the following processing: obtain evaluation information for a specific location in the virtual space, and based on the results of the evaluation information, use the first image to change the image at the specific location in the virtual space. Figure 1 Part (c) of FIG. 1 is a diagram showing an example of such an image change. This part shows the image in part (b) being mapped to Figure 1 An example of a three-dimensional object is shown in part (a).

[0059] The image captured in the real world may not reflect the entire object in the virtual space. Therefore, it is possible to change the image of a part of the object instead of the entire image based on the image captured in the real world. Figure 2 This is a conceptual diagram showing a scenario in which a portion of an object image is changed. Figure 2 Part (a) shows an image captured in the real world (with Figure 1 Compared with the image shown in part (b), the range of the image is narrower). Figure 2 Part (b) shows a scene where a portion of the object's image is modified based on the image shown in part (a) (a scene where the image shown in part (a) is mapped onto the object). Furthermore, by modifying a portion of the object's image in this way, the processing load can be reduced compared to modifying the entire object's image.

[0060] Furthermore, when changing an image, an image subjected to predetermined image processing (described later) may be used.

[0061] [Structure of image processing system]

[0062] Figure 3 1 is a diagram showing a schematic configuration of an image processing system according to the first embodiment. Figure 3 As shown, image processing system 1 includes user systems 10 (user terminals, VR display devices, goggle-type devices) and image processing server 20 (image processing device), which are connected via a network NW (communication line) such as the Internet or the cloud. User systems 10 may be one or more. Image processing server 20 can access database 30 (image storage device).

[0063] [Structure of image processing server]

[0064] Figure 4 2 is a diagram showing the structure of the image processing server 20 (image processing device). Figure 4As shown, the image processing server 20 is provided with a processor 22 (processor), a ROM 24 (ROM: Read Only Memory), and a RAM 26, and is connected with a database 30. The processor 22 provides a virtual space information acquisition section 22A, a specific position evaluation section 22B, an evaluation information acquisition section 22C, an evaluation information judgment section 22D, an image change section 22E, and an output control section 22F. The image processing server 20 uses the processor 22 to manage the position, shape, sound within the virtual space, lighting (brightness, chroma, etc.), and changes thereof, and the like, of the objects in the virtual space, and to perform the change of the image as described later.

[0065] The processor 22 is constituted by, for example, various processors or circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a FPGA (Field Programmable Gate Array), a PLD (Programmable Logic Device), and the like. When these processors or circuits execute software (program), the computer (for example, various processors or circuits constituting the processor and / or their combination) to be executed reads a code of the software stored in a non-transitory and tangible recording medium such as the ROM 24, and the computer refers to the software. The software stored in the non-transitory and tangible recording medium includes an image processing program (program for causing the computer to execute the image processing method according to the present application) and data used when the software is executed, which are related to the present application. The code can also be recorded in a non-transitory and tangible recording medium such as a flash ROM or an EEPROM (Electronically Erasable and Programmable Read Only Memory) instead of the ROM 24. In addition, the "non-transitory and tangible recording medium" does not include an intangible recording medium such as a carrier wave or a propagated signal itself. The RAM 26 is used as a temporary storage area or a work area when processing is performed using the software.

[0066] Details of the processing using the processor 22 having the above-described structure will be described later.

[0067] [Information recorded in the database]

[0068] [Recording of the image and related information]

[0069] The database 30 has various non-temporary and tangible recording media such as a magneto-optical recording medium, a semiconductor recording medium, and a control device thereof, and can record and output various information necessary for management of the virtual space and image processing. Therefore, information of the virtual space, at least including three-dimensional position information of the virtual space, is recorded in the database 30. The three-dimensional position information of the virtual space is preferably information corresponding to three-dimensional position information in the real world, and, for example, is preferably determined by a combination of latitude, longitude, altitude, or coordinates equivalent thereto (three-dimensional orthogonal coordinates, etc.). The three-dimensional position information of the virtual space can have information of shape, color, brightness, pattern, etc., and can be attached with some image or texture in an initial state (in a state where image change described later is not performed). In addition, the information of the virtual space is preferably information imitating the real world in terms of shape, color, brightness, etc., but does not necessarily have to be strictly the same, as long as a user of the image processing system 1 can recognize the same as the real world.

[0070] In addition, in the database 30, an image photographed in the real world and / or an image obtained by performing image processing on an image photographed in the real world is recorded in correspondence with the three-dimensional position. These images are one way of "first image" and "first image group" in the present application, and, as described later in detail, the first image is used to change an image at a specific position in the virtual space. In addition, the database 30 preferably records relevant information such as photographer information or user comments in association with the images.

[0071] Figure 5 is a diagram illustrating an example of an image and relevant information recorded in the database 30. As shown in Figure 5 In the database 30, three-dimensional position information of the virtual space, an image of the real world corresponding to the position, photographer information of the image, and user evaluation of the image, etc. are recorded in association. The image processing server 20 (processor) can refer to the information to select a first image (an image of the real world corresponding to a specific position, an image used to change an image of the virtual space), and can present the information to the user. The user can refer to the presented information to perform an operation of selecting a first image. Hereinafter, a case where information is recorded in association will be described.

[0072] [Association of Position and Image]

[0073] "Three-dimensional position" is three-dimensional position information in the virtual space corresponding to three-dimensional position information in the real world. In addition, in the present application, the three-dimensional position is preferably determined by a combination of latitude, longitude, altitude, or coordinates equivalent thereto (three-dimensional orthogonal coordinates, etc.), and is preferably determined by a combination of latitude, longitude, altitude, or coordinates equivalent thereto (three-dimensional orthogonal coordinates, etc.). Figure 5In the example of FIG. 1, the three-dimensional position is shown by a number, but it is preferable to associate the number with three-dimensional coordinates (e.g., latitude, longitude, altitude). Also, it is preferable to associate the three-dimensional position with a name of a place (or a name of a facility or a sign, etc.). The "comprehensive evaluation" is an evaluation (evaluation value) of the position or the place, and can be calculated, for example, by numerically processing one or more evaluation information and statistically processing (total, weighted total, average, etc.). Also, a plurality of images can be recorded for the first position, and a plurality of evaluation information can be recorded for one image.

[0074] In the example of FIG. 1, the three-dimensional position is shown by a number, but it is preferable to associate the number with three-dimensional coordinates (e.g., latitude, longitude, altitude). Also, it is preferable to associate the three-dimensional position with a name of a place (or a name of a facility or a sign, etc.). The "comprehensive evaluation" is an evaluation (evaluation value) of the position or the place, and can be calculated, for example, by numerically processing one or more evaluation information and statistically processing (total, weighted total, average, etc.). Also, a plurality of images can be recorded for the first position, and a plurality of evaluation information can be recorded for one image. Figure 5 In the example of FIG. 1, the three-dimensional position is shown by a number, but it is preferable to associate the number with three-dimensional coordinates (e.g., latitude, longitude, altitude). Also, it is preferable to associate the three-dimensional position with a name of a place (or a name of a facility or a sign, etc.). The "comprehensive evaluation" is an evaluation (evaluation value) of the position or the place, and can be calculated, for example, by numerically processing one or more evaluation information and statistically processing (total, weighted total, average, etc.). Also, a plurality of images can be recorded for the first position, and a plurality of evaluation information can be recorded for one image.

[0075] [Photographer information]

[0076] The image is associated with the photographer information of the image. The "identification information" is, for example, a name or an ID of the photographer, or a nickname, etc., and the attribute of the photographer (age, sex, whether professional or amateur, etc.) can be associated with the identification information. Also, the "biological characteristic information" is, for example, the heart rate, blood pressure, respiration rate, brain wave, etc. of the photographer at the time of photographing the image, and this biological characteristic information can be used as a clue to determine whether the photographer was excited or moved at the time of photographing the image, or whether the photographer was in a calm, tranquil mood, etc. The "comment" is, for example, a comment of the photographer, such as "I was very moved!", "At this place, it is recommended to view the waterfall from the back", "It is best to view this place on a sunny day", etc., and this comment can be used as a clue to determine whether the position or the place as an object is a high-score place (or the above-mentioned highlight experience area or highlight experience space).

[0077] [Evaluation information]

[0078] The evaluation (evaluation information) of the user can be recorded in association with the position in the virtual space or the image. The evaluation information can be composed of one item, or can be composed of a plurality of items (N items in the example of FIG. 1) like a vector in a multi-dimensional characteristic amount space. Also, a plurality of items and a comprehensive evaluation can be included. The evaluation information preferably includes at least one of an evaluation of the "place itself" and an evaluation of the image. In addition, the evaluation information can include a comment of the user (e.g., "I was also very moved!", "I want to go there!", etc.). Figure 5

[0079] Figure 6 ​FIG is a diagram showing a scene in which an image and evaluation information are displayed. Figure 6 In the figure, the image, the photographer, the photographer's comments, and the results of user reviews are shown ( Figure 6 In the example, the comprehensive evaluation and the distribution of the evaluations are shown). The image processing server 20 (processor 22) can select high-scoring places (high-scoring locations) based on the evaluation information and Figure 6 It will be displayed as shown. Figure 6 When an image (first image) is selected in the virtual space, as will be described in detail later, the image processing server 20 changes the image at the location (specific position) shown in the image and presents it to the user (displaying it on the display 204 of the goggles 200). In the image processing system 1, by presenting the image and evaluation information to the user, the image processing server 20 enables sharing of feelings and encourages a desire to visit the site.

[0080] [Structure of user system]

[0081] Figure 7 1 is a diagram showing the configuration of a user system in the first embodiment. Figure 7 As shown, the user system 10 includes goggles 200 (server connection device, goggles-type device) and a router 300. The goggles 200 are connected to the network NW via the router 300. The goggles 200 and the router 300 can be connected via wired communication or wireless communication such as Wi-Fi (registered trademark). Furthermore, the user system 10 may include terminal devices such as smartphones, tablet computers, and personal computers. These terminal devices can be used for uploading images, as described later.

[0082] [Structure of goggles]

[0083] Figure 8 2 is a diagram showing the structure of the goggles 200. Figure 8 As shown, the goggles 200 include a processor 202, a display 204, an operating unit 206, a flash ROM 208, a RAM 210, a wireless communication interface 212, a microphone 214, and a speaker 216. They communicate with the image processing server 20 via a router 300 and a network NW. Specifically, the goggles 200 function as a server connection device and a virtual space display device, and can be configured as a VR goggle-type device (VR: Virtual Reality) worn (mounted) on the user's head. The goggles 200 may also include a memory card and / or motion sensors (such as acceleration sensors and angular velocity sensors). Furthermore, the user system 10 may include motion sensors separate from the goggles 200. These motion sensors can be worn on the user's head, torso, or limbs, for example, and can be used to detect changes in the user's position or posture.

[0084] As with the description of the processor 22 of the image processing server 20, the processor 202 of the goggles 200 can be configured using a CPU, a GPU, or the like, and perform processing such as connection with the image processing server 20 or display of a virtual space. In performing the processing, a program or data stored in the flash ROM 208 can be used, and the RAM 210 can be used as a temporary storage area or a work area. Connection with the image processing server 20 is made via the wireless communication interface 212. Also, the user can perform operations necessary for browsing (accessing) a virtual space or the like via the operation section 206. In addition, the display 204 can display a three-dimensional virtual space, an image, information related to an image, or the like.

[0085] Also, the user can perform an operation of inputting a voice via the microphone 214, the speaker 216, listening to a sound within a virtual space, and talking with another user connected (participating) to the virtual space, by the goggles 200. The goggles 200 (the processor 202) can output a message to the user via these devices. In addition, the processor 202 can perform voice recognition on a voice input via the microphone 214.

[0086] [Flowchart of image processing method]

[0087] Figure 9 、 Figure 10 is a flowchart showing the processing of the image processing method in the first embodiment.

[0088] [Connection with image processing system]

[0089] If the processing starts, the processor 202 sets the goggles 200 to a virtual space connection mode, and connects to the image processing system 1 (the image processing server 20) (step S100). The trigger (opportunity) of the connection can be, for example, detection of an operation on the operation section 206, detection of a specific motion by the motion sensor 218, recognition of a voice input to the microphone 214 (at this time, it is assumed that the processor 202 of the goggles 200 has a voice recognition function), or the like.

[0090] [Specification of image selection condition]

[0091] The processor 22 (processor) specifies a condition for selecting the first image by an operation of the user or automatically without the operation of the user (step S110). The operation of the user is, for example, an operation on the operation section 206 and / or input of a voice to the microphone 214. The processor 202 of the goggles 200 can perform voice recognition on the operation content input via the microphone 214, and transmit the result thereof to the image processing server 20.

[0092] The selection conditions of the 1st image are as follows: for example, a high user rating, an image that is the latest (posted within 7 days or the like), an image that is of a specific season (spring, summer, autumn, winter), a time period (morning, daytime, evening, night or the like), or weather (sunny, rainy, snowy or the like), an image that is taken by a photographer with a specified identification information and / or attribute, or the like. These conditions can be specified in the following process. Also, the user can specify the conditions in advance or additionally using a device (for example, a smartphone or a personal computer) other than the goggles 200 that is capable of connecting to the image processing system 1.

[0093] [Specification of the Access Place in the Virtual Space]

[0094] The processor 22 determines the place that the user will access in the virtual space (acquires a specific position in the virtual space) by the user's operation or automatically without the user's operation. The user's operation can be an operation of inputting the name or a keyword (for example, "mountain", "sea", "random" or the like) of a specific place or a landmark, or an operation of selecting a candidate prompted by the image processing server 20, and the user can operate the operation section 206 and / or input a voice to the microphone 214. Also, the processor 22 determines the access place by the following methods: selection from places with a high user rating based on the evaluation information in the database 30 (refer to Figure 5 ), determination based on a keyword input by the user, and random determination, or the like. In addition, the access place can be specified or changed in the following process.

[0095] [Output of the Information of the Virtual Space]

[0096] The processor 22 outputs the information of the virtual space of the specified access place (step S130). Specifically, the processor 22 transmits the three-dimensional shape or an object or the like of the specified access place to the user system 10 and displays it on the display 204 of the goggles 200. As described above, the information of the virtual space becomes information that imitates the real world, and can have information of color or brightness in addition to the three-dimensional position information and the three-dimensional shape information. The information of the virtual space can have voice information in addition to these visual information. The goggles 200 can output sound from the speaker 216.

[0097] The goggles 200 detect the user's movement or change in posture at a determined timing by the motion sensor 218 and transmits the detected information to the image processing server 20. The processor 22 of the image processing server 20 outputs the information of the virtual space corresponding to the information according to the reception of the information.

[0098] Figure 11 is a diagram showing an example of the information displayed on the display 204. Figure 11The (a) part of FIG. 1 shows an image taken in the real world, Figure 11 The (b) part of FIG. 1 shows a scene of a virtual space corresponding to the real world (in a state where the "image change" described later is not performed). The information of the virtual space includes three-dimensional position information and a three-dimensional shape, and has color, brightness, and texture, and the like. In addition, in the (b) part of FIG. 1, the texture of water or the surface of a rock is expressed with a fill pattern, but images of rocks, soil, water, trees, grass, and the like can also be mapped on the three-dimensional shape. Figure 11

[0099] In addition, when the scene of the virtual space is displayed, an avatar of the user himself or another user can be displayed. Also, it can be configured so that the user can select whether or not to display the avatar. The image processing server 20 can perform processing related to the display or movement of the avatar in the virtual space.

[0100] [Acquisition of specific position and acquisition of evaluation information]

[0101] The processor 22 acquires a position (specific position) that the user is interested in in the virtual space (step S140). The processor 22 can acquire the specific position based on the three-dimensional position of the access site described above and the posture information of the user (which can be detected by the motion sensor 218). Then, the processor 22 refers to the database 30 to acquire evaluation information related to the evaluation of the specific position (step S150).

[0102] [Judgment of evaluation information]

[0103] The processor 22 judges whether or not the evaluation information acquired for the specific position satisfies a predetermined criterion (step S160). If the specific position is a high-scored place (for example, a popular tourist spot), the processor 22 can judge it as "satisfying the predetermined criterion". Specifically, the processor 22 can judge at least one of the number of photographs of the first image for the specific position, the photographer or the number of the first image, the number of affirmative evaluations, the user's comment, and the biometric information of the photographer associated with the first image as "the predetermined criterion". If the evaluation information satisfies the criterion, the process will proceed to step S170, and if the evaluation information does not satisfy the criterion, the process will proceed to step S220. The processor 22 can take, for example, "the number of images is 100 or more" or "the user's comprehensive evaluation is four stars or more out of five (see the example of FIG. 2)" as the criterion. Figure 6

[0104] [Change of image at specific position]

[0105] [Processing before changing image]

[0106] ​​The evaluation value of the number of times of photographing the first image and the like described above is an example of an evaluation value calculated based on the evaluation information recorded in the database 30, and if the evaluation value is equal to or higher than a threshold value, the processor 22 changes the image at the specific position in the virtual space (step S200). In addition, before changing the image, the processor 22 can perform the processing described below.

[0107] The processor can change the image in accordance with detection of a trigger (the case where "Yes" in step S170), and can set a predetermined operation or a predetermined action performed by the user as the "trigger". The "predetermined operation" or "predetermined action" can be detected by the operation section 206, the microphone 214, and the motion sensor 218. In addition, utterance of a specific sentence is also included in the predetermined operation or action.

[0108] The processor 22 selects the first image based on the conditions described above (step S180). In addition, as described later (with reference to the description of step S320 regarding the selection of the first image), the processor 22 can determine the photographing place of the image and select the first image based on the determined photographing place. Figure 14

[0109] The processor 22 can perform image processing on the selected first image (step S190). This is because the impression received by different users can be different even for the same image. Furthermore, the processor 22 can adjust the degree of change or the degree of image processing in accordance with the attributes of the user. Here, the "attributes" are one or more of the age, sex, and preferences of the user, for example. It is known that the spectral visibility decreases with age, and in particular, the spectral visibility of light of short wavelengths (wavelengths of about 400 nm to 500 nm, violet to blue) decreases (according to JIS S 0031 and the like), and for example, if the user is an elderly person, it can be considered to perform image processing that emphasizes violet or blue.

[0110] If the determination and processing of steps S170 to S190 are completed, the processor 22 changes the image at the specific position in the virtual space using the selected first image (step S200). Figure 12 is a diagram showing the scene of changing the image. Figure 12 The (a) part of shows the scene of the virtual space before the change, and the dotted line part is the region of the changed image (the place where the user enjoys or focuses; the specific position), Figure 12 The (b) part of shows the selected first image (the image photographed in the real world). Figure 12 The (c) part of shows the scene after the change of the image (assuming the case where the first image is mapped on the three-dimensional shape of the virtual space). In addition, the "access place" in this example includes a wide range of a waterfall, a flat terrain in the vicinity thereof, and a cliff, and the "specific position" is the top of the cliff. ​

[0111] If the above-described change is made to all areas or a wide range of the virtual space, the processing load can become high, and it can be difficult or impossible to implement in reality. However, according to the first embodiment, even in this case, by changing the image at the specific position, it is possible to reflect the scene in the real world into the virtual space, and it is possible to reduce the processing load.

[0112] [Continuous change of image]

[0113] When the above-described change is made, depending on the relationship between the information of the virtual space and the image used for the change (the first image) or the manner of the change, the user can feel uncomfortable. For example, it can be considered that the user feels uncomfortable when the virtual space and the first image are greatly different in color or brightness or discontinuously change near the boundary of the changed portion. Therefore, in the first embodiment, when the image of the virtual space is changed, the processor 22 can continuously change the image of the virtual space corresponding to the photographed range of the first image and the image of the virtual space corresponding to the area outside the photographed range in space or time.

[0114] Figure 12 The (d) part of FIG. 1 shows an example in which the boundary of the area in which the image is changed is blurred to change smoothly in the virtual space and the first image ("a manner of changing continuously in space"). In addition to blurring the image, the brightness or color can be changed continuously. Also, the processor 22 can adjust the shape or size of the image as needed to connect the boundary smoothly.

[0115] The above-described "continuous change" can also be "continuous change in time". For example, it can be considered that the processor 22 does not immediately (instantaneously) change the entire image, but gradually (in a certain period of time) approaches the image of the real world in time. At this time, the situation in which the user approaches the specific position can be taken as a trigger to start the change. Also, the "continuous change" can also be "continuous change with respect to sound". For example, near the boundary of the area in which the image is changed, it can be considered that the processor 22 reduces the volume of the sound associated with the establishment of the virtual space and the sound associated with the first image.

[0116] When the image of the virtual space is changed, the processor 22 can display the comment of the photographer of the image used for the change (the first image) or the comment of another user, evaluation information, or the like.

[0117] In addition, the presence or absence of the change or the content thereof can differ for each user. Also, the "change" can include the case in which the virtual space in which the image is not set is newly set with the image in addition to the case in which the image of the initial state is changed.

[0118] [Input of comment on image]

[0119] The user who has viewed the image in the virtual space that has been changed in this manner can enter a comment on the image. The user can enter a comment in the form of, for example, selecting a number of "stars" or a numerical value indicating a rating, selecting "like," or entering text. The user can enter a comment via the operation unit 206 or microphone 214 of the goggles 200. The processor 22 associates the entered comment with the image at the specific position (the first image) and records it in the database 30 (with the first image). Figure 5 corresponding to the "Comments" in the "Evaluation Information").

[0120] [Information about places you visit, etc.]

[0121] The processor 22 can present information related to a place or a specific location (hereinafter referred to as “visited place, etc.”) visited by the user in the virtual space to the user (step S210 ). Figure 13 is a diagram showing an example of display of such information, showing the state after the above-mentioned changes are made ( Figure 12 In the state shown in part (d) of FIG, the states of buttons 900 to 904 are displayed. Figure 13 In this example, button 900 links to a website (e.g., a website of a local government, tourism association, or facility operator) that provides information on the geographical features, history, and local customs of a potential visit destination (a real-world location corresponding to a virtual world destination). Button 902 links to a website of a souvenir shop. Button 904 links to a website of a travel agency that offers travel products targeting the virtual world destination. This information allows users to gain a deeper understanding of the virtual world destination and cultivate their interest in or interest in the destination. Furthermore, it increases users' willingness to visit the real-world destination (the potential visit destination).

[0122] [Acquisition and Recording of the First Image]

[0123] The acquisition and recording of the image (image captured in the real world; first image) used in the above-mentioned change will be described. Figure 14 This is a flowchart showing the process of acquiring and recording images.

[0124] The processor 22 obtains images uploaded to the image processing system 1 (image processing device) by multiple users and related information about the images (step S300). The "related information" is, for example, the date and time of shooting of the image or the information of the shooter (see Figure 5 The user can connect a digital camera, a smartphone, a personal computer, etc. to the image processing server 20 via, for example, the goggles 200 or the router 300, and upload images and related information.

[0125] Processor 22 can then access publicly available websites or databases via network NW and retrieve images and related information from these websites or databases (step S310). These "websites" include blogs and various SNS (social networking services or social networking sites). The images retrieved in steps S300 and S310 constitute a first image group.

[0126] Furthermore, regarding the order in which steps S300 and S310 are executed, any step may be executed first, and these steps may be repeated. Furthermore, the processor 22 may only execute one of steps S300 and S310. Furthermore, regarding the acquisition of images and related information, it is assumed that usage permissions, etc., will be acquired separately or have already been acquired.

[0127] [Determination of filming location, etc.]

[0128] The processor 22 determines the shooting location of the real-world image included in the first image group (step S320). The determined shooting location is associated with the image and recorded in the database 30 (refer to Figure 5 (For example, in step S180, the first image is selected based on the determined shooting location. For images in the first image group that have latitude and longitude information (an example of three-dimensional location information) for the shooting location provided by a positioning system such as the GPS (Global Positioning System), the processor 22 can determine the shooting location based on the latitude and longitude information of each image. Furthermore, even if the image lacks latitude and longitude information provided by GPS or other means, the processor 22 can determine the shooting location using at least one of feature extraction and pattern matching between images. This feature extraction is performed using machine learning methods. The machine learning method used to determine the shooting location is not particularly limited; for example, neural networks such as CNNs (Convolutional Neural Networks) or DNNs (Deep Neural Networks) or other deep learning algorithms can be used. Alternatively, a trained model can be used, which has been trained to output the shooting location when fed with learning data that pairs images with shooting locations. Furthermore, pattern matching can be performed between the images included in the first image group and images for which the shooting location is known.

[0129] In addition to the shooting location, the processor 22 preferably also determines at least one of the shooting azimuth angle, the shooting elevation angle, and the shooting magnification. Once this information is determined, the processor 22 can select the first image from the first image group based on the shooting location and at least one of the shooting azimuth angle, the shooting elevation angle, and the shooting magnification (step S180 described above).

[0130] [Image processing for acquired images]

[0131] The processor 22 determines whether to perform image processing on the captured image (step S330). If the determination is "yes," the processor 22 performs image processing on the captured image (step S340). This is because image processing can potentially produce an image closer to the photographer's intended image, even if the captured image differs from the photographer's image. Examples of image processing include, but are not limited to, adjustments to one or more of the following: image movement, enlargement or reduction, rotation, focus, noise, hue, brightness, contrast, and outline. Furthermore, the processor 22 may determine the image processing based on or independently of user instructions.

[0132] The processor 22 can perform image processing corresponding to the change in shooting time. For example, the processor 22 can perform such image processing when the photographer realizes that "the photo was actually taken during the day, but the intended photo was taken at dusk" or "the photo was taken on a cloudy day, but the intended photo was taken on a sunny day." Figure 15 is a diagram showing an example of image processing corresponding to a change in the shooting time. Figure 11 The state shown in part (a) of the image (taken under bright but cloudy conditions).

[0133] Figure 15 Parts (a) to (c) of the image processing are examples of images that have been processed to create the effect of being taken during a clear day, at dusk, and at a clear night. Figure 15 In the example shown, image processing is performed to change the color and brightness of the sky in the image, and objects (a bright sun, a setting sun, stars) are placed to represent a blue sky, a sunset sky, or a starry sky. However, processor 22 may also change the brightness or color tone of the ground based on the virtual shooting time. For example, to create a dusk scene, the ground or water surface may be painted red.

[0134] In addition, the processor 22 can determine whether to perform the above-described image processing based on, for example, "whether the user instructed image processing when uploading the image" or "whether the original intention of the photographer is included in the comment of the photographer". The processor 22 can analyze the comment of the photographer and infer the "original intention of the photographer" using a machine learning method when making such a determination. Furthermore, the method for image processing is not particularly limited, and for example, a machine learning method such as GAN (Generative Adversarial Networks) can be used.

[0135] In the image processing system 1, by such image processing, it is possible to easily share the image in the mind of the photographer or the emotion of the photographer among users. In addition, if image processing is performed, the processor 22 can perform the above-described change on the image on which image processing has been performed as the first image (step S200).

[0136] The processor 22 records the image (the acquired image itself or the image on which image processing has been performed) in association with the related information in the database 30 (step S350). Furthermore, if the specific location is changed, the processor 22 changes the image according to the change (if "Yes" in step S220, proceed to step S150). If the access place is changed, the processor 22 returns to step S130 and repeats the above-described processing.

[0137] As described above, according to the image processing system 1 according to the first embodiment, it is possible to stimulate the desire to visit or revisit a place in the real world, share the emotion related to a specific place with others, and the like, and thus it is possible to improve the value of the VR experience.

[0138] [Modified Example]

[0139] In the above-described first embodiment, the eyewear 200 is used in connection with the image processing server 20 and image display, but a smart phone or a tablet terminal, a personal computer, or the like can be used instead of the eyewear-type device. When using these devices, the sense of immersion in the virtual space is reduced compared to the case of the eyewear-type device, but since the field of view is not blocked, the user can easily access the virtual space. Furthermore, these devices generally have a character input function using a keyboard or a touch screen, and thus it is easy to input a comment or the like.

[0140] The above describes the embodiments and modified examples of the present application, but the present application is not limited to the above-described modes, and various modifications can be made.

[0141] Explanation of Symbols

[0142] 1 - image processing system, 10 - user system, 20 - image processing server, 22 - processor, 30 - database, 200 - goggles, 202 - processor, 204 - display, 206 - operation section, 212 - wireless communication interface, 214 - microphone, 216 - speaker, 218 - motion sensor, 300 - router, 900 - button, 902 - button, 904 - button.

Claims

1. An image processing apparatus comprising: a processor; and a memory storing a program for causing the processor to execute, the processor referring to the memory, processes: acquiring information of a virtual space including three-dimensional position information; acquiring a specific position in the virtual space; acquiring evaluation information related to evaluation of the specific position; judging the evaluation information based on a predetermined reference; and changing an image at the specific position in the virtual space using a first image of a real world corresponding to the specific position according to a result of the judging.

2. The image processing apparatus according to claim 1, wherein the processor changes the image at the specific position in the virtual space using the first image when an evaluation value calculated based on the evaluation information is equal to or greater than a threshold value.

3. The image processing apparatus according to claim 1 or 2, wherein the processor judges using at least one of the following information as the reference: a number of photographings of the first image for the specific position; a photographer of the first image for the specific position; a number of positive evaluations for the specific position; a comment of a user of the image processing apparatus for the specific position; and biometric information of the photographer associated with the first image for the specific position.

4. The image processing apparatus according to claim 1 or 2, wherein the processor selects the first image based on a condition designated by a user of the image processing apparatus or a condition determined by the processor.

5. The image processing apparatus according to claim 1 or 2, wherein the processor determines a photographing place of an image of a real world included in a first image group, and selects the first image based on the photographing place.

6. The image processing apparatus according to claim 5, wherein the processor determines the photographing place using at least one of feature quantity extraction using a method of machine learning and pattern matching between images.

7. The image processing apparatus according to claim 6, wherein the processor processes: determining at least one of an azimuth of photographing, an elevation of photographing, and a magnification of photographing in addition to the photographing place; and selecting the first image from the first image group based on at least one of the azimuth of photographing, the elevation of photographing, and the magnification of photographing and the photographing place.

8. The image processing apparatus according to claim 1 or 2, wherein in the changing, the processor changes, spatially or temporally, an image of the virtual space corresponding to a photographing range of the first image and an image of the virtual space corresponding to a region other than the photographing range continuously.

9. The image processing apparatus according to claim 1 or 2, wherein the processor changes in response to a case where a user of the image processing apparatus has performed a predetermined operation or a predetermined action.

10. The image processing apparatus according to claim 1 or 2, wherein The processor adjusts the degree of the change according to an attribute of a user of the image processing apparatus.

11. The image processing apparatus according to claim 10, wherein The processor adjusts the degree of the change using one or more of an age, a gender, and a preference of the user as the attribute.

12. The image processing apparatus according to claim 1 or 2, wherein The processor performs the change using an image uploaded to the image processing apparatus by a user of the image processing apparatus as the first image.

13. The image processing apparatus according to claim 1 or 2, wherein The processor accesses a website available to the public via a network and uses an image acquired from the website as the first image for the change.

14. The image processing apparatus according to claim 1 or 2, wherein The processor sets a sound corresponding to the specific location based on a sound associated with the first image when performing the change.

15. The image processing apparatus according to claim 1 or 2, wherein The processor prompts information related to the specific location to a user of the image processing apparatus.

16. The image processing apparatus according to claim 15, wherein The processor prompts at least one of the following information to the user as the information related to the specific location: an explanation of a place in the real world corresponding to the specific location, that is, a visit candidate; a website introducing the visit candidate; a visit address of a travel agency handling a travel product for the visit candidate; and a visit address of a sales store selling a souvenir of the visit candidate.

17. The image processing apparatus according to claim 1 or 2, wherein The processor performs the change using an image subjected to image processing on the first image as the first image.

18. The image processing apparatus according to claim 17, wherein The processor performs the image processing on the first image based on an instruction of a user of the image processing apparatus.

19. The image processing apparatus according to claim 1 or 2, further comprising an image storage apparatus that stores the first image, The processor performs the change based on an image selected from the first image stored in the image storage apparatus.

20. The image processing apparatus according to claim 1 or 2, wherein The processor determines the specific location based on an output of a sensor worn by a user of the image processing apparatus and detecting a position and a posture of the user.

21. The image processing apparatus according to claim 1 or 2, wherein The processor displays the virtual space on a goggle-type apparatus worn by a user of the image processing apparatus and including a display apparatus.

22. An image processing method executed by an image processing apparatus including a processor and a memory storing a program for causing the processor to execute, in the image processing method, The processor performs the following processing with reference to the memory: acquiring information of a virtual space including three-dimensional position information; acquiring a specific position in the virtual space; acquiring evaluation information related to an evaluation of the specific position; judging the evaluation information based on a predetermined criterion; and changing an image at the specific position in the virtual space using a first image, which is an image of a real world corresponding to the specific position, according to a result of the judging.

Images