Information processing system, information processing method, information processing program, and mobile

Abstract

To provide an information processing system which presents incompatibility to a user through complexity of a time-varying image, an information processing method, an information processing program, and a mobile.SOLUTION: In an information processing system comprising at least one processor, the at least one processor executes a program to implement the following steps. The program executes: a target image acquisition step of acquiring a target image which varies with the passage of time; a calculation step of calculating a complexity parameter indicating complexity of time-variations of the target image based on the target image; and a display control step of displaying a predetermined three-dimension object based on the complexity parameter in such a manner that the more complicated the variation of the target image is per unit time, the more spatially finely a surface state of the three-dimensional object is distorted in relative to a reference state. The reference state is a surface state of the three-dimensional object corresponding to a case where the complexity parameter is a reference value.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, an information processing program, and a moving body.

Background Art

[0002] Patent Document 1 discloses an image generation device, a display system, and a program capable of obtaining an image that changes according to the surrounding environment of a moving body. The image generation device includes an information generation unit (parameter generation unit) that generates environment information (such as a complexity parameter) representing the degree of change in the surrounding environment of the moving body, and an image generation unit that controls image processing according to the environment information to generate an image that changes according to the environment information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is still room for improvement in the technology for presenting a sense of discomfort to the user through the complexity of an image that changes over time.

Means for Solving the Problems

[0005] According to one aspect of the present invention, an information processing system is provided. This information processing system includes at least one processor. The at least one processor is configured to execute a program such that the following steps are performed. In the target image acquisition step, a target image that changes over time is acquired. In the calculation step, based on the target image, a complexity parameter indicating the complexity of the temporal change of the target image is calculated. In the display control step, based on the complexity parameter, as the change per unit time of the target image becomes more complex, a three-dimensional object is displayed such that the surface state of a predetermined three-dimensional object is spatially finely distorted compared to the reference state. The reference state is the surface state of the three-dimensional object corresponding to the case where the complexity parameter is a reference value.

[0006] According to such a configuration, it is possible to provide a better technique for presenting a sense of discomfort to the user through the complexity of an image that changes over time.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic matters shown in the following embodiments can be combined with each other.

[0009] Incidentally, the program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium readable by a computer, may be provided so as to be downloadable from an external server, or may be provided so that the program is launched on an external computer to realize its functions on a client terminal (so-called cloud computing).

[0010] Also, in this embodiment, the "section" may include, for example, a combination of hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in this embodiment, various types of information are handled, and these types of information are represented, for example, by physical values of signal values representing voltage and current, the high and low of signal values as a set of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0011] Also, a circuit in a broad sense is a circuit realized by appropriately combining at least a circuit, circuitry, a processor, a memory, and the like. That is, it includes an application specific integrated circuit (ASIC), programmable logic devices (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), and the like.

[0012] 1. Hardware Configuration In this section, the hardware configuration will be described.

[0013] <Information Processing System 1> FIG. 1 is a configuration diagram showing Information Processing System 1. Information Processing System 1 includes, for example, a vehicle 2 as a moving object.

[0014] Vehicle 2 is configured to be operable by a passenger U who is an example of a user. Vehicle 2 may be a general passenger car or an industrial vehicle such as a forklift or a towing tractor. Vehicle 2 includes a seat 3, an in-vehicle camera 4 which is an example of an imaging device, a display unit 5, a car audio 6, and an information processing device 7.

[0015] Seat 3 is configured such that passenger U can sit on it. Seat 3 may include an air jack 31. Air jack 31 is an example of a perception information output unit and is configured to output tactile information as perception information that can be perceived by passenger U other than visually. Specifically, air jack 31 is configured to deform the shape (for example, unevenness) of the contact surface between passenger U and seat 3 by supplying and exhausting air from the outside. By supplying and exhausting air to air jack 31, the change in the seating feeling of seat 3 felt by passenger U is output as tactile information.

[0016] In-vehicle camera 4 is configured to generate an in-vehicle image IM1 by imaging at least a part of the surroundings of vehicle 2. In-vehicle camera 4 of the present embodiment is a so-called front camera configured to image the front of vehicle 2. Note that in-vehicle camera 4 may be a so-called back camera configured to image the rear of vehicle 2 when traveling backward, or a combination of these.

[0017] The in-vehicle image IM1 can change over time. The time change of the in-vehicle image IM1 can be caused by the relative movement of an object included in the imaging range of the in-vehicle camera 4 with respect to the vehicle 2. For example, the time change of the in-vehicle image IM1 tends to increase as the speed of the vehicle 2 increases. Also, the time change of the in-vehicle image IM1 tends to increase according to the amount of change in the traveling direction of the vehicle 2 due to a sharp curve or the like. Further, the time change of the in-vehicle image IM1 tends to increase in a region corresponding to a wall surface or the like, which is an object around the vehicle 2, as the wall surface is closer. Therefore, the time change of the in-vehicle image IM1 tends to increase when relatively high attention is required of the passenger U in operating the vehicle 2.

[0018] The display unit 5 is configured to display a screen of a graphical user interface (GUI) operable by the user. The display unit 5 can be implemented using a display device such as a CRT display, a liquid crystal display, an organic EL display, and a plasma display. The display unit 5 may be integrated with the vehicle 2, for example, an instrument panel, or may be externally attached, for example, a display for car navigation or a smartphone attachable to the vehicle 2. The display unit 5 can be configured to display information visible to the passenger U. For example, the display unit 5 is configured to be able to display the three-dimensional object Ob2 described later.

[0019] The car audio 6 is configured to output auditory information perceptible to the passenger U through hearing. The auditory information can include information regarding various sounds audible to the user, such as sound effects, warning sounds, and music. The air jack 31 and the car audio 6 are each an example of a perception information output unit other than the display unit 5.

[0020] The information processing device 7, the air jack 31, the in-vehicle camera 4, the display unit 5, and the car audio 6 are configured to be communicable through a telecommunication line. In one embodiment, the information processing system 1 consists of one or more devices or components. For example, if it consists only of the information processing device 7, the information processing system 1 can be the information processing device 7. Hereinafter, these components will be described.

[0021] <Information processing device 7> FIG. 2 is a block diagram showing the hardware configuration of the information processing device 7. The information processing device 7 includes a communication unit 71, a storage unit 72, at least one processor 73, and an HMI device 74, and these components are electrically connected via a communication bus 70 inside the information processing device 7.

[0022] The communication unit 71 preferably uses wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), and wired LAN network communication. However, wireless LAN network communication, mobile communication such as 3G / LTE / 5G, and BLUETOOTH (registered trademark) communication may be included as necessary. That is, it is more preferable to implement it as a set of these multiple communication means. That is, the information processing device 7 may communicate various information from the outside via the communication unit 71 and the network.

[0023] The storage unit 72 stores various information defined by the foregoing description. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the information processing device 7 executed by the processor 73, or as a memory such as a random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to the calculation of the program. The storage unit 72 stores various programs, variables, and the like related to the information processing device 7 executed by the processor 73.

[0024] The processor 73 processes and controls the overall operations related to the information processing device 7. The processor 73 is, for example, a Central Processing Unit (CPU) not shown in the figure. By reading a predetermined program stored in the storage unit 72, the processor 73 realizes various functions related to the information processing device 7. That is, the information processing by the software stored in the storage unit 72 is specifically realized by the processor 73, which is an example of hardware, and can be executed as each functional unit included in the processor 73. These will be described in more detail in the next section. Note that the processor 73 is not limited to being single, and may be implemented to have a plurality of processors 73 for each function, or a combination thereof.

[0025] The processor 73 is configured to be able to acquire information from the in-vehicle camera 4 or other devices. The processor 73 is configured to be able to acquire various information by reading various information stored in a storage area that is at least part of the storage unit 72 and writing the read information to a working area that is at least part of the storage unit 72. The storage area is, for example, an area implemented as a storage device such as an SSD in the storage unit 72. The working area is, for example, an area implemented as a memory such as a RAM. Note that the acquisition by the processor 73 includes acquiring the output results of each functional unit included in the processor 73.

[0026] The processor 73 is configured to be able to display various information as a display processing unit. The information can be presented to the user via the display unit 5 or other devices. In such a case, for example, the processor 73 controls the display unit 5 to display visual information such as a screen, an image including a still image or a moving image, an icon, a message, etc. The processor 73 may generate only the rendering information for displaying the visual information on the display unit 5. Note that the processor 73 may present the output information to the user without going through the display unit 5 or other device users.

[0027] The HMI device 74 is a Human-Machine Interface device. The HMI device 74 may be included in the housing of the vehicle 2 or may be externally attached. For example, the HMI device 74 may be integrated with the display unit 5 and implemented as a touch panel. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, instead of the touch panel, a switch button, a mouse, a QWERTY keyboard, a voice recognition device, a gesture detection device, a gaze detection device, a biological signal detection device, an imaging device, etc. may be adopted. That is, the HMI device 74 receives the operation input made by the user. The HMI device 74 transfers, as a response, a signal corresponding to the operation input to the processor 73 via the communication bus 70. The processor 73 can execute predetermined control and calculations as necessary. It can also be said that the HMI device 74 includes an input unit configured to be able to receive input from the user.

[0028] 3. Example of Information Processing In this section, the information processing executed in the information processing system 1 described above will be explained.

[0029] 3.1. Flow of Information Processing FIG. 3 is an activity diagram showing an example of the flow of information processing executed in the information processing system 1. Note that the information processing may include any exception processing not shown. Exception processing includes interruption of the information processing and omission of each process. The selection or input performed in the information processing may be based on the operation of the user or may be automatically performed without depending on the operation of the user.

[0030] [Activity A1] First, in activity A1, the processor 73 acquires model data. The model data is a three-dimensional model of a predetermined specific shape. The three-dimensional model represented as the model data may include any shape that can be represented in three-dimensional space, such as a sphere, an ellipse, a cube, a rectangular parallelepiped, etc. Further, the three-dimensional model represented as the model data is not limited to a model having a curved surface, and may be a shape that can be pseudo-three-dimensionally represented by observing a two-dimensional planar object from a direction other than the normal direction of the object. In the present embodiment, the model data is represented as a spherical three-dimensional model. The model data may be stored in advance in the storage unit 72 or may be arbitrarily imported by the passenger U.

[0031] [Activity A2] Next, in activity A2, the processor 73 acquires the environmental image IM0. The environmental image IM0 may be stored in advance in the storage unit 72 or may be arbitrarily imported by the passenger U. The environmental image IM0 may be a still image that does not change with time or a dynamic image that changes with time. The dynamic image may include a movable object as an element that changes with time. The movable object is configured to be able to move freely in the dynamic image based on a predetermined algorithm such as physical calculation.

[0032] [Activity A3] Next, in activity A3, the processor 73 generates a three-dimensional object Ob2 represented by displaying the environmental image IM0 on the surface of the three-dimensional model defined by the model data. In the present embodiment, in activity A3, the surface state of the three-dimensional object Ob2 generated regardless of the complexity parameter described later is set as the reference state of the three-dimensional object Ob2. The surface state of the three-dimensional object Ob2 is not limited to the shape of the surface of the three-dimensional object Ob2, and may be defined by any information that defines the state of the surface, such as the texture (especially the three-dimensional feeling) of the environmental image IM0 displayed on the surface.

[0033] [Activity A4] Next, in Activity A4, the processor 73 causes the display unit 5 to display the display image IM3. The display image IM3 is configured to display the latest three-dimensional object Ob2. In Activity A4, the processor 73 causes the display image IM3 including the three-dimensional object Ob2 to be displayed.

[0034] [Activity A5] Next, in Activity A5, the processor 73 acquires the in-vehicle image IM1 from the in-vehicle camera 4. The processor 73 acquires the latest image (an example of the first target image) included in the in-vehicle image IM1 and the image immediately before the latest image in the in-vehicle image IM1 (an example of the second target image). The difference between these images can suggest the temporal change of the in-vehicle image IM1. Then, the processor 73 determines whether the environmental image IM0 acquired in Activity A2 includes the movable object Ob1b. The determination can be made based on, for example, the format of the environmental image IM0.

[0035] [Activity A6] When it is determined that the environmental image IM0 includes the movable object Ob1b, the process proceeds to activity A6, and the processor 73 updates the state of the movable object Ob1b in the environmental image IM0 based on the change in the in-vehicle image IM1. For example, the processor 73 introduces a force to the movable object Ob1b in response to the movement of pixels highly likely to represent the same object in the in-vehicle image IM1, and performs a physical operation on the movable object Ob1b based on the introduced force, thereby moving the movable object Ob1b in the environmental image IM0. The processor 73 updates the state such as the position and orientation of the movable object Ob1b, which has changed as a result of such a physical operation, as the state of the latest movable object Ob1b. Thereby, the environmental image IM0 displayed on the surface of the three-dimensional object Ob2 changes. In activity A6, it is preferable that the update of the state of the movable object Ob1b is performed using a change different from the change of the environmental image IM0 used for calculating the complexity parameter described later. In other words, the environmental image IM0 may be a video that changes over time. According to such a configuration, since the environmental image IM0 that changes over time is distorted according to the complexity parameter, it is possible to easily notice the change in the complexity of the in-vehicle image IM1 from the shape change of the three-dimensional object Ob2. Thereafter, the process proceeds to activity A7. When it is determined that the environmental image IM0 does not include the movable object Ob1b, the process of activity A6 is omitted.

[0036] [Activity A7] Next, at activity A7, the processor 73 calculates a complexity parameter based on the in-vehicle image IM1 obtained at activity A5. The complexity parameter indicates the complexity of the temporal change of the in-vehicle image IM1. For example, the processor 73 calculates the complexity parameter based on the difference between the latest image in the in-vehicle image IM1 and an image in the in-vehicle image IM1 that is a predetermined period before a certain image (e.g., the current image). According to such a configuration, the difference between the image a predetermined period before and the current image can be visually grasped. When the time element is expressed using frames, the predetermined period is specifically, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 frames, and it may be within the range between any two of the numerical values exemplified here. In the present embodiment, the processor 73 calculates the complexity parameter based on the difference between the latest image in the in-vehicle image IM1 and the image immediately before the latest image in the in-vehicle image IM1 (in other words, the image one frame before). Specifically, the processor 73 identifies the number of pixels that are different between the latest image and the immediately preceding image, and calculates the complexity parameter from a relational expression that defines the relationship between the number of pixels and the complexity parameter. In other words, the processor 73 calculates the amount of change from the second target image to the first target image as the complexity parameter. According to such a configuration, the time required to calculate the complexity parameter can be shortened, and the complexity of the change in the in-vehicle image IM1 can be reflected in the three-dimensional object Ob2 in real time. The complexity parameter in the present embodiment is a scalar value. The relational expression is defined such that the complexity parameter increases as the number of pixels increases. When the in-vehicle image IM1 does not change, the number of pixels becomes almost 0, and as the change in the in-vehicle image IM1 becomes more complex, the number of pixels increases. Therefore, the number of substantially different pixels correlates with the complexity of the change in the in-vehicle image IM1. Note that the calculation mode of the complexity parameter is not limited to this and is arbitrary. For example, a pre-trained learned model that takes a plurality (e.g., two) of images as input and outputs the complexity parameter for the plurality of images may be used to calculate the complexity parameter.

[0037] [Activity A8] Thereafter, the processor 73 receives, for example, an image processing specification based on an operation by the user, and when the image processing specification is input, the process proceeds to activity A8. In activity A8, the processor 73 performs image processing to change the environmental image based on the complexity parameter. For example, as the complexity parameter increases, the processor 73 adds an effect such that the environmental image IM0 becomes easier to distinguish from the background color of the display image IM3. Thereby, the state of the environmental image IM0 is updated. Thereafter, the process proceeds to activity A9. Note that when there is no image processing specification, the process of activity A9 is omitted.

[0038] [Activity A9] In activity A9, the processor 73 generates a modulated image IM2 based on the complexity parameter calculated in activity A7. The modulated image IM2 is an image having, as the main component of the frequency spectrum, a value corresponding to the complexity parameter by performing two-dimensional Fourier transform. The modulated image IM2 of the present embodiment is a pseudo-pattern image obtained by forming a random pattern so as to have the main frequency component. RGB values are assigned to each pixel position of the modulated image IM2 so as to form the random pattern. Note that the modulated image IM2 may be a pattern image having periodicity corresponding to the main frequency component. Further, the modulated image IM2 is not limited to a color image to which RGB values are assigned, and may be a monochrome image or a binary image. In the process of distorting the surface state of the three-dimensional object Ob2 described later, in order to introduce a more delicate distortion, it is preferable that the image is in a format in which pseudo-continuous values can be set for each pixel, such as a color image or a monochrome image.

[0039] [Activity A10] Next, at activity A10, the processor 73 performs processing on the three-dimensional object Ob2 generated at activity A3. The processing is a process of distorting the surface state of the three-dimensional object Ob2 based on the complexity parameter. As a result, the processor 73 causes the surface state of the three-dimensional object Ob2 to be spatially finely distorted compared to the reference state as the change per unit time of the in-vehicle image IM1 becomes complex, for example, as the complexity parameter increases. The reference state is the surface state of the three-dimensional object Ob2 corresponding to the case where the complexity parameter is the reference value. According to such a configuration, since the time change of the in-vehicle image IM1 can be visually recognized by the passenger U as the distortion of the three-dimensional object Ob2, the discomfort associated with the time change that is difficult to notice from the in-vehicle image IM1 itself for the passenger U can be made more noticeable from the shape change of the three-dimensional object Ob2. In the present embodiment, the basic state is the surface state of the three-dimensional object Ob2 generated at activity A3. The surface state of the three-dimensional object Ob2 takes a state in which no distortion is introduced, that is, a state in which the complexity parameter is 0.

[0040] Here, an example of a process for distorting the surface state of the three-dimensional object Ob2 based on the complexity parameter will be described. First, similar to the case of the environmental image IM0, the processor 73 unfolds the modulation image IM2 on the surface of the three-dimensional object Ob2. As a result, the position of the modulation image IM2 (for example, the pixel position) and the position on the surface of the three-dimensional object Ob2 are associated with each other. Consequently, the environmental image IM0 and the modulation image IM2 are associated with each other through the surface of the three-dimensional object Ob2. Then, the processor 73 performs a process of distorting the surface state of the three-dimensional object Ob2 based on the state (for example, RGB value) of the modulation image IM2 associated with the surface of the three-dimensional object Ob2. Such a process may be, for example, an image process for deforming the environmental image IM0 developed on the surface of the three-dimensional object Ob2, a deformation process for deforming the surface of the three-dimensional object Ob2 itself, or a combination of these. In particular, the processor 73 executes a process for distorting the three-dimensional sense of the surface of the three-dimensional object Ob2 based on the complexity parameter (in this embodiment, the modulation image IM2 generated based on the complexity parameter). For example, the processor 73 performs a process of increasing the amount of deformation of the surface of the three-dimensional object Ob2 as the scalar value that can be calculated from the RGB values of the modulation image IM2, for example, the luminance, is larger.

[0041] When the state of the movable object Ob1b in the environmental image IM0 is updated in Activity A6, or when image processing on the environmental image IM0 is executed in Activity A8, the processor 73 changes the environmental image IM0 developed on the surface of the three-dimensional object Ob2 to the updated latest environmental image IM0, and then executes the above processing on the three-dimensional object Ob2. In other words, the processor 73 distorts the environmental image IM0 on which image processing has been performed in Activity A8 and is displayed on the surface of the three-dimensional object Ob2 based on the complexity parameter, thereby displaying the three-dimensional object Ob2. According to such a configuration, in addition to the change of the three-dimensional object Ob2, since the environmental image itself further changes according to the complexity of the temporal change of the in-vehicle image IM1, it is possible to make it easier for the passenger U to notice the complexity of the change of the in-vehicle image IM1.

[0042] As described above, the in-vehicle image IM1 is an image captured by the in-vehicle camera 4 provided in the vehicle 2 that can be operated by the passenger U. By performing the above processing on the three-dimensional object Ob2 using such an in-vehicle image IM1, the passenger U can recognize the change in complexity that is difficult to notice from the in-vehicle image IM1 during the operation of the vehicle 2 as a change in the surface state of the three-dimensional object Ob2. Therefore, it is possible to prompt the passenger U to pay attention to a slight change in the situation around the vehicle 2.

[0043] Thereafter, the processor 73 ends this information processing. Note that the processor 73 may further repeat the processing of Activities A5 to A10 while the ignition of the vehicle 2 is on, and end this information processing when the ignition is turned off.

[0044] 3.2. Specific Example of Information Processing When a Still Image is Used as the Environmental Image Next, a specific example of the above information processing when the still image IM0a is used as the environmental image IM0 will be described. FIG. 4 is a diagram showing the display mode of the three-dimensional object Ob2 when the still image IM0a and the in-vehicle image IM1a are used.

[0045] First, the still image IM0a used in this section will be described. As shown in FIG. 4, the still image IM0a is an image that does not change with time and is an example of the environmental image IM0 that does not include the movable object Ob1b. For example, the still image IM0a is an image defined by a periodic pattern. According to such a configuration, the complexity of the change in the target image can be visually recognized as the distortion of the periodic pattern. The still image IM0a in the present embodiment is a two-dimensional planar image including the background region R1a and the stationary object Ob1a. The background region R1a is a region that defines the background color of the still image IM0a. The background color is configured to match, for example, the basic display color of the display unit 5. In the present embodiment, the background color is black. The stationary object Ob1a indicates the minimum unit of the pattern to be drawn. The stationary object Ob1a has a color different from the background color, specifically, a color distinguishable from the background color. In the present embodiment, the color of the stationary object Ob1a is a yellow-based color that is easily distinguishable from the black background color. The stationary object Ob1a is arranged on a lattice having periodicity along each of the first direction and the second direction intersecting the first direction. Specifically, the stationary object Ob1a is arranged on a rectangular lattice defined by the first direction (vertical direction) and the second direction (horizontal direction). The stationary object Ob1a of the present embodiment has symmetry about the first direction and the second direction, for example, mirror symmetry. Thereby, since the symmetry of the entire still image IM0a is improved, the distortion of the periodic pattern can be made easier to visually recognize.

[0046] Next, a three-dimensional object Ob2 generated based on the still image IM0a used in this section will be described. In the present embodiment, the processor 73 displays the still image IM0a on the surface of the spherical model data such that the vertical direction of the still image IM0a coincides with the longitude direction on the spherical surface and the horizontal direction of the still image IM0a coincides with the latitude direction of the spherical surface. The processor 73 displays the three-dimensional object Ob2 configured in this manner within the display image IM3. Note that the display image IM3 includes an external region R2 corresponding to the outside of the three-dimensional object Ob2. The external region R2 is a region corresponding to the background of the display image IM3. The color of the external region R2 is arbitrary, but for example, it is preferably the same as the color of the background region R1a of the still image IM0a included in the three-dimensional object Ob2. According to such a configuration, since the three-dimensional object Ob2 is likely to blend into the display image IM3, the stationary object Ob1a can be emphasized, and thus the change in the still image IM0a can be made more noticeable.

[0047] Next, an in-vehicle image IM1a as the in-vehicle image IM1 used in this section will be described. The in-vehicle image IM1a shows an image in front of the vehicle 2 when the vehicle 2 is stopped at a stop line. Here, for convenience of explanation, the objects included in the in-vehicle image IM1a are in a substantially stationary state, and the in-vehicle image IM1a is in a state with almost no change. In this case, since there are almost no regions where the pixels are different between the latest in-vehicle image IM1a and the immediately preceding in-vehicle image IM1a, the complexity parameter becomes almost zero. Based on such a substantially zero complexity parameter, the processor 73 generates a modulated image IM2a, which is an example of the modulated image IM2. The main frequency components included in the modulated image IM2a are sufficiently larger than the frequency components representing the periodicity of the still image IM0a. The modulated image IM2 in the present embodiment is configured such that RGB values are assigned for each pixel coordinate. The RGB values change almost continuously at adjacent pixel coordinates. As a result, in the modulated image IM2a, regions of the same or similar colors are formed with a domain size corresponding to the main frequency components corresponding to the complexity parameter.

[0048] Based on the RGB values of the modulated image IM2a thus generated, the processor 73 distorts the still image IM0a displayed at the position corresponding to the surface of the three-dimensional object Ob2 displayed in the display image IM3. In the present embodiment, the processor 73 causes the three-dimensional object Ob2 to be displayed such that the surface state is distorted by changing the three-dimensional effect of the still image IM0a displayed on the surface of the three-dimensional object Ob2. According to such a configuration, the surface state of the three-dimensional object Ob2 can be made visible to the passenger U due to the distortion of the environmental image, and it is possible to facilitate the discrimination of changes in the surface state of the three-dimensional object Ob2. Specifically, the processor 73 sets a parameter corresponding to the height based on the RGB values of the modulated image IM2 for each position on the surface of the three-dimensional object Ob2, and distorts the still image IM0a on the three-dimensional object Ob2 so as to represent the pseudo unevenness on the three-dimensional object Ob2 defined thereby in the still image IM0a developed on the surface of the three-dimensional object Ob2.

[0049] Note that, as described above, since the main frequency component included in the modulated image IM2a is sufficiently larger than the frequency component representing the periodicity of the still image IM0a, almost no deformation of the still image IM0a occurs. Therefore, hereinafter, for convenience of explanation, the surface state of the three-dimensional object Ob2 is substantially equal to the reference state. Therefore, the display mode of the three-dimensional object Ob2 shown in FIG. 4 is treated as the reference state of the three-dimensional object Ob2 when the still image IM0a is used.

[0050] Next, the display mode of the three-dimensional object Ob2 when using the in-vehicle image IM1b, which is an in-vehicle image IM1 with a more complex change per unit time compared to the above in-vehicle image IM1a, will be described. FIG. 5 is a diagram showing an example of the display mode of the three-dimensional object Ob2 when using the in-vehicle image IM1b. As shown in FIG. 5, the in-vehicle image IM1b is an imaging result of the in-vehicle camera 4 when the vehicle 2 is traveling on a road with a narrower road width compared to the in-vehicle image IM1a. The distance between the vehicle 2 and surrounding objects, for example, a wall surface, a street tree, a utility pole, etc., is short. Therefore, even if traveling at the same speed, the complexity of the change per unit time of the in-vehicle image IM1b is more intense compared to the complexity of the change per unit time of the in-vehicle image IM1a. As a result, the value of the complexity parameter calculated from the in-vehicle image IM1b is larger than the complexity parameter calculated from the in-vehicle image IM1a. As a result, the modulated image IM2b obtained from the in-vehicle image IM1b forms a pattern having a spatially finer domain size compared to the modulated image IM2a. Since the pattern of the modulated image IM2b is sufficiently shorter than the length of the outer peripheral edge of the three-dimensional object Ob2 (here, the circumference of the sphere), the surface state of the three-dimensional object Ob2 is finely distorted based on the modulated image IM2b. As a result, the regular pattern displayed on the surface of the three-dimensional object Ob2 shown in FIG. 5 is displayed as being bent.

[0051] Next, the display mode of the three-dimensional object Ob2 when using the in-vehicle image IM1c, which is an in-vehicle image IM1 with an even more complex change per unit time compared to the above in-vehicle image IM1b, will be described. FIG. 6 is a diagram showing an example of the display mode of the three-dimensional object Ob2 when using the in-vehicle image IM1c. The in-vehicle image IM1c has a higher traveling speed of the vehicle 2 compared to the case where the in-vehicle image IM1b is captured on a narrow road width. Therefore, in this case, the complexity parameter calculated from the in-vehicle image IM1c becomes even larger than when calculated from the in-vehicle image IM1b. Therefore, the processor 73 generates a modulated image IM2c with an even finer spatial pattern compared to the modulated image IM2. As a result, the surface state of the three-dimensional object Ob2 is distorted even more finely so that the pattern of the original still image IM0a cannot be recognized.

[0052] In this way, the processor 73 distorts the surface state of the three-dimensional object Ob2 by changing the three-dimensional effect of the still image IM0a displayed on the surface of the three-dimensional object Ob2. As the change of the in-vehicle image IM1 becomes more complex, the passenger U can visually recognize the change in complexity that is difficult to notice from the in-vehicle images IM1a to IM1c themselves through the fact that the still image IM0a displayed on the surface of the three-dimensional object Ob2 is finely distorted from the state of a periodic pattern to a chaotic state. In particular, because it is an intuitive change in shape, the passenger U can intuitively notice the change in the shape of the three-dimensional object Ob2 even while the vehicle 2 is running. Therefore, the change in the situation during running can be intuitively conveyed to the passenger U.

[0053] 3.3. Specific Example of Information Processing When Using a Dynamic Image as the Environmental Image Next, a specific example of the above information processing when using the dynamic image IM0b as the environmental image IM0 will be described.

[0054] First, the dynamic image IM0b used in this section will be described. FIG. 7 is a diagram showing an example of the state of change of the dynamic image IM0b. The dynamic image IM0b includes a leaf-shaped object as the movable object Ob1b and a black region that constitutes the background of the object. The movable object Ob1b is configured to move the dynamic image IM0b in conjunction with the change of an image such as the in-vehicle image IM1, particularly the movement of an arbitrary object in the image. The movement of the movable object Ob1b can be controlled based on an arbitrary mechanical simulation.

[0055] In this section, image processing is further performed in activity A8. FIG. 8 is a diagram showing changes in the dynamic image IM0b when a process of superimposing an optical effect is used as image processing based on the complexity parameter. As shown in FIG. 9, the processor 73 superimposes an optical effect of a color different from the background colors of the environment image IM0 and the display image IM3, for example, white, based on the complexity parameter on the dynamic image IM0b. By adopting such a color, changes in the three-dimensional object Ob2 can be emphasized more strongly. The range over which the light is superimposed becomes wider as the complexity parameter increases. Specifically, the optical effect is configured to diffuse from the center toward the outer edge as the complexity parameter increases.

[0056] Next, changes in the display mode of the three-dimensional object Ob2 based on changes in the complexity parameter will be described. FIG. 9 is a diagram showing an example of the display mode of the three-dimensional object Ob2 when the in-vehicle image IM1d is used as the in-vehicle image IM1. The in-vehicle image IM1d is an image captured when the vehicle 2 is stationary, similar to the in-vehicle image IM1a. Therefore, the complexity parameter calculated from the in-vehicle image IM1d is relatively low. Accordingly, almost no external force is applied to the movable object Ob1b included in the dynamic image IM0b. As a result, the movable object Ob1b remains in a state of being diffused throughout the dynamic image IM0b, and the white light is only slightly superimposed in the center. Also, since the complexity parameter is small, the modulated image IM2d as the generated modulated image IM2 is also an image having a relatively large domain size. Therefore, the dynamic image IM0b displayed on the three-dimensional object Ob2 is maintained in a state with almost no distortion. On the other hand, the effect of the slightly superimposed white light colors the central portion of the dynamic image IM0b with a color different from the background color of the display image IM3. As a result, the central portion of the three-dimensional object Ob2 appears faintly under the influence of the white light. That is, the distortion of the dynamic image IM0b is more emphasized by the image processing performed on the dynamic image IM0b.

[0057] FIG. 10 is a diagram showing an example of a display mode of the three-dimensional object Ob2 when the in-vehicle image IM1e is used as the in-vehicle image IM1. The in-vehicle image IM1e has a greater complexity of change than the in-vehicle image IM1d. As a result, the complexity parameter increases. Along with this, white light is slightly superimposed on the dynamic image IM0b as a whole. Also, due to the movement of an object such as a signboard in the in-vehicle image IM1e, an external force is applied to the movable object Ob1b in the dynamic image IM0b. As a result, the movable object Ob1b moves so as to be biased toward the upper center region. Further, the processor 73 generates a modulated image IM2e having a domain size finer than the modulated image IM2d based on the complexity parameter calculated from the in-vehicle image IM1e. As a result, on the surface of the three-dimensional object Ob2, in addition to the fine pattern corresponding to the movable object Ob1b, a three-dimensional pattern corresponding to the pattern of the modulated image IM2e appears with respect to the white light spreading over the entire dynamic image IM0b.

[0058] FIG. 11 is a diagram showing an example of a display mode of the three-dimensional object Ob2 when the in-vehicle image IM1f is used as the in-vehicle image IM1. The in-vehicle image IM1f has an even greater complexity of change than the in-vehicle image IM1e. As a result, the complexity parameter further increases from the situation shown in FIG. 10. Since the white light superimposed on the dynamic image IM0b is saturated, there is almost no change from the state shown in FIG. 10. The same applies to the movement of the movable object Ob1b. On the other hand, the modulated image IM2f is configured to have a finer pattern than the modulated image IM2e as the complexity parameter increases. As a result, the surface of the three-dimensional object Ob2 is in a state of pseudo-forming finer irregularities compared to FIGS. 10 and 11, similar to the pattern of the modulated image IM2f for the white light of the dynamic image IM0b.

[0059] 3.4. An example of the display mode of the three-dimensional object when model data of a shape other than a sphere is used In the above embodiment, the three-dimensional object Ob2 was generated using spherical model data. However, the present invention is not limited to this, and any shape can be adopted. FIG. 12 is a diagram showing an example of the display mode of the three-dimensional object Ob2 according to the complexity parameter when the three-dimensional object Ob2 is generated using cubic model data. In this section, as in the previous section, image processing is performed such that white light is superimposed on the environmental image IM0. As shown in FIG. 12, when the complexity parameter is small, almost the entire three-dimensional object Ob2 assimilates to the background color, so that the distortion on its surface is difficult to see. On the other hand, as a result of superimposing the white light as the complexity parameter increases, the surface of the three-dimensional object Ob2 becomes a color different from the background color, and a distorted pattern similar to the pattern of the modulation image IM2 is displayed on the surface of the cube.

[0060] Further, the model data may be planar. FIG. 13 is a diagram showing an example of the display mode of the three-dimensional object Ob2 when planar model data is used. In this case, the processor 73 can display the three-dimensional object Ob2 pseudo-spatially in the three-dimensional space by displaying in the display image IM3 a state in which the planar model data is observed obliquely. In this case, the color and unevenness of the position of the three-dimensional object Ob2 change according to the pattern of the modulation image IM2 obtained from the complexity parameter, so that the three-dimensional object Ob2 can be displayed such that the surface state of the three-dimensional object Ob2 is spatially finely distorted compared to the reference state.

[0061] 3.5. An example of the display mode of the three-dimensional object when changing the shape of the surface of the three-dimensional object Next, an example of the display mode of the three-dimensional object Ob2 will be described when the three-dimensional object Ob2 is displayed such that the surface state of the three-dimensional object Ob2 is spatially finely distorted compared to the reference state by changing the shape of the three-dimensional object Ob2. FIG. 14 is a diagram showing an example of the display mode of the three-dimensional object Ob2 when the shape of the three-dimensional object Ob2 is changed based on the complexity parameter. As shown in FIG. 14, as the complexity parameter increases, the pattern of the modulation image IM2 becomes finer, and accordingly, the unevenness of the three-dimensional object Ob2 becomes finer. This is obtained by converting the scalar values such as the RGB values of the modulation image IM2 into the height on the three-dimensional object Ob2 and forming the unevenness of the surface of the three-dimensional object Ob2 according to the height. In this way, the processor 73 can display the three-dimensional object Ob2 such that the shape of the three-dimensional object Ob2 is distorted according to the complexity parameter. According to such a configuration, the complexity of the change in the target image can be intuitively conveyed to the user from the shape of the three-dimensional object Ob2 itself.

[0062] 3.6. Information processing capable of changing the display mode according to the complexity parameter The processor 73 may perform information processing such that the surface state of the three-dimensional object Ob2 is distorted in a different manner according to the complexity parameter. FIG. 15 is an activity diagram showing another example of the information processing executed by the information processing system 1. As shown in FIG. 15, the processor 73 performs the processes of activities A1 to A9 described together with FIG. 3 to obtain the latest environmental image IM0 and the modulation image IM2.

[0063] After that, in activity A101, based on the calculated complexity parameter, the processor 73 determines a display condition indicating how to distort the three-dimensional object Ob2. The display condition may include any information related to the display mode of the three-dimensional object Ob2 based on the complexity parameter, such as, for example, a threshold for changing the distortion method, a threshold of the complexity parameter for determining whether to introduce image processing for the environmental image IM0, etc. For example, the display condition is defined such that when the complexity parameter is less than a predetermined threshold, the surface state of the three-dimensional object Ob2 is distorted in a first mode, and when the complexity parameter is greater than or equal to the threshold, the display mode of the three-dimensional object Ob2 is distorted in a second mode different from the first mode. The first mode is, for example, a mode of distorting the surface of the three-dimensional object Ob2 in a mode of changing the three-dimensional effect of the environmental image IM0 displayed on the surface of the three-dimensional object Ob2 as described in Section 3.2 or the like. The second mode is, for example, a mode of distorting the surface of the three-dimensional object Ob2 in a mode of changing the shape of the three-dimensional object Ob2 as described in Section 3.5. According to such a configuration, an appropriate change in the surface state can be caused according to the degree of complexity, and the complexity in a wide range can be displayed in a manner that is easy to visually recognize.

[0064] Next, in activity A102, based on the generated modulated image IM2, the processor 73 changes the display mode of the three-dimensional object Ob2 according to the display condition determined in activity A101. According to such a configuration, for example, even in a case where the change in the in-vehicle image IM1 becomes so large that the change in the surface state according to the first mode becomes difficult to visually recognize as it approaches the threshold, the change in the complexity of the change in the in-vehicle image IM1 can be visually recognized as the change in the surface state according to the second mode. Therefore, the complexity of the change in the in-vehicle image IM1 in a wider range can be visually recognized as the change in the surface state of the three-dimensional object Ob2. After that, the processor 73 ends this information processing.

[0065] 3.7. In addition to changing the display mode of the three-dimensional object Ob2, information processing that can drive the car audio 6 or the like based on the complexity parameter will be described. Based on the calculated complexity parameter, in addition to changing the display mode of the three-dimensional object Ob2, the processor 73 causes the car audio 6 and the air jack 31 as the perceptual information output units to output perceptual information in different modes according to the complexity parameter. According to such a configuration, since the user can perceive the complexity of the change in the environment around the moving body not only visually but also by other means, it becomes easier to notice the change in the surrounding complexity. FIG. 16 is an activity diagram showing an example of information processing in which, in addition to changing the display mode of the three-dimensional object Ob2, the output mode of the perceptual information output unit can be changed. As shown in FIG. 16, the processor 73 performs the processes of activities A1 to A10 described together with FIG. 3. As a result, the calculation of the complexity parameter, the generation of the modulated image IM2, and the change in the display mode of the three-dimensional object Ob2 are performed.

[0066] Thereafter, the processor 73 changes the acoustic information reproduced from the car audio 6 based on the complexity parameter. Here, an example of the process for conveying the complexity of the change in the environmental image IM0 based on the sound will be described. In activity A101, the processor 73 acquires acoustic information regarding the sound being reproduced using the car audio 6. The acoustic information includes the frequency spectrum of the sound output as the sound over time.

[0067] Next, in activity A102, the processor 73 performs harmony analysis on the sound. As a result, the processor 73 identifies the harmony of the sound output as the sound and the phonemes that constitute the dissonant sound with respect to the harmony.

[0068] Next, at activity A103, the processor 73 superimposes phonemes that constitute the dissonance identified at activity A102 on the sound output from the car audio 6 based on the complexity parameter. For example, when the complexity parameter is a threshold value, the processor 73 superimposes the dissonance, and increases the magnitude of the phoneme as the complexity parameter increases. In this way, by making the sound during playback become dissonant according to the complexity, the change in the complexity of the in-vehicle image IM1 can be made noticeable to the passenger U.

[0069] Also, after activity A10, the processor 73 performs the process of activity A201. In the present embodiment, activity A101 and activity A201 are performed in parallel, but it is not limited to this. At activity A201, the processor 73 changes the pressure in the air jack 31 based on the complexity parameter. Specifically, when the complexity parameter becomes equal to or greater than a preset value, the processor 73 introduces air into the air jack 31 to change the seating feeling of the seat 3. Thereby, the change in the complexity of the surrounding situation can be tactually conveyed to the passenger U. Note that the air jack 31 is not limited to the seat 3, and may be provided, for example, at a portion where the passenger U such as a steering wheel comes into contact.

[0070] After that, the processor 73 ends this information processing.

[0071] 4. Others The above embodiment is merely an example. For example, the above embodiment can be configured as follows.

[0072] The imaging device is not limited to an optical one that captures the surroundings of the vehicle 2 like the in-vehicle camera 4. For example, the imaging device may be configured to image the entire area as a point cloud such as LiDAR (Light Detection And Ranging), or may measure the information around the vehicle 2 acoustically like a sonar and perform imaging by visually reconstructing the surrounding situation based on the measurement result.

[0073] The user is not limited to the passenger U. For example, when the vehicle 2 is configured to be remotely operable, the user may be a remote operator of the vehicle 2 instead of the passenger U. In this case, the display unit 5 can be displayed on a display panel visible to the remote operator.

[0074] The vehicle 2 is not limited to an automobile, and may include vehicles used as public transportation such as trains, bullet trains, linear motor cars, etc., and vehicles assumed to run other than on public roads such as go-karts and industrial vehicles. Further, the moving body may include devices movable other than on the ground such as flying automobiles, airplanes, ships, etc. Further, the moving body is not limited to those moving in the real space, and may include those moving in the digital space, for example, in a virtual space such as a metaverse. That is, the target image is not limited to an image in the real space, and may be an image in the virtual space.

[0075] The target image is not limited to the in-vehicle image IM1, and may include any image that can change over time, such as a video of a video camera formed by the user, a movie, an anime video, etc.

[0076] The information processing device 7 is not limited to being integrally mounted on the vehicle 2, and may be, for example, a portable terminal such as the passenger U's smartphone, or an information processing device provided outside the vehicle 2 such as a cloud server.

[0077] The information processing device 7 may be in an on-premises form or a cloud form. As the information processing device 7 in the cloud form, for example, in the form of SaaS (Software as a Service), cloud computing, the above-described functions and processes may be provided.

[0078] In the above embodiment, the information processing device 7 performs various storage and controls, but instead of the information processing device 7, a plurality of external devices may be used. That is, various information and programs may be distributed and stored in a plurality of external devices using blockchain technology or the like.

[0079] The above-described embodiment is not limited to the information processing system 1, and may be an information processing method or an information processing program. The information processing method includes each step of the information processing system 1. The information processing program causes at least one computer to execute each step of the information processing system 1.

[0080] The above-described information processing system 1 etc. may be provided in each of the following aspects.

[0081] (1) An information processing system including at least one processor, wherein the at least one processor is configured to execute a program such that the following steps are performed. In a target image acquisition step, a target image that changes over time is acquired. In a calculation step, based on the target image, a complexity parameter indicating the complexity of the temporal change of the target image is calculated. In a display control step, based on the complexity parameter, as the change per unit time of the target image becomes more complex, the three-dimensional object is displayed such that the surface state of a predetermined three-dimensional object is spatially finely distorted compared to a reference state, and the reference state is the surface state of the three-dimensional object corresponding to the case where the complexity parameter is a reference value.

[0082] According to such a configuration, since the user can visually recognize the temporal change of the target image as the distortion of the three-dimensional object, it is possible to make it easier for the user to notice the discomfort associated with the temporal change that is difficult to notice from the target image itself from the shape change of the three-dimensional object.

[0083] (2) The information processing system according to (1) above, wherein in the display control step, the three-dimensional object is displayed such that the surface state is distorted by changing the three-dimensional effect of a predetermined environmental image displayed on the surface of the three-dimensional object.

[0084] According to such a configuration, since the surface state of the three-dimensional object can be visually recognized by the user due to the distortion of the environmental image, it is possible to easily determine the change in the surface state of the three-dimensional object.

[0085] (3) In the information processing system according to (2) above, the environmental image is an image defined by a periodic pattern, system.

[0086] According to such a configuration, the complexity of the change in the target image can be visually recognized as the distortion of a periodic pattern.

[0087] (4) In the information processing system according to (2) or (3) above, in the further image processing step, further, based on the complexity parameter, image processing is performed to change the environmental image, and in the display control step, based on the complexity parameter, the environmental image on which the image processing has been performed and which is displayed on the surface of the three-dimensional object is distorted to display the three-dimensional object, system.

[0088] According to such a configuration, in addition to the change in the three-dimensional object, since the environmental image itself further changes according to the complexity of the temporal change of the target image, it is possible to make it easier for the user to notice the complexity of the change in the target image.

[0089] (5) In the information processing system according to any one of (2) to (4) above, the environmental image is a video that changes over time, system.

[0090] According to such a configuration, since the environmental image that changes over time is distorted according to the complexity parameter, it is possible to make it easier to notice the change in the complexity of the target image from the shape change of the three-dimensional object.

[0091] (6) In the information processing system according to any one of (1) to (5) above, in the display control step, a system that displays the three-dimensional object such that the shape of the three-dimensional object is distorted according to the complexity parameter.

[0092] According to such a configuration, the complexity of the change in the target image can be intuitively conveyed to the user from the shape of the three-dimensional object itself.

[0093] (7) In the information processing system according to any one of (1) to (6) above, in the display control step, a system that displays the three-dimensional object so as to change the mode of distorting the surface state of the three-dimensional object based on the complexity parameter.

[0094] According to such a configuration, for example, even when the change in the surface state according to the first mode becomes difficult to visually recognize due to the complexity of the change in the target image increasing up to near the threshold value, the complexity of the change in the target image can be visually recognized as the change in the surface state according to the second mode. Therefore, the complexity of the change in the target image over a wider range can be visually recognized as the change in the surface state of the three-dimensional object.

[0095] (8) In the information processing system according to (7) above, the mode of distorting the surface state of the three-dimensional object includes changing the three-dimensional feeling of the environmental image displayed on the surface of the three-dimensional object and changing the shape of the three-dimensional object.

[0096] According to such a configuration, an appropriate change in the surface state can be caused according to the degree of complexity, and the complexity over a wide range can be displayed in a mode that is easy to visually recognize.

[0097] (9) In the information processing system according to any one of (1) to (8) above, in the calculating step, the complexity parameter is calculated based on the difference between a first target image that is the latest image in the target image and a second target image that is an image in the target image and is a predetermined period before the first target image.

[0098] According to such a configuration, the difference between the image from a predetermined period before and the current image can be visually grasped.

[0099] (10) In the information processing system according to (9) above, the second target image is the image immediately before the first target image in the target image, and in the calculating step, the amount of change from the second target image to the first target image is calculated as the complexity parameter.

[0100] According to such a configuration, the time required for calculating the complexity parameter can be shortened, and the complexity of the change in the target image can be reflected in the three-dimensional object in real time.

[0101] (11) In the information processing system according to any one of (1) to (10) above, the target image is a video captured by an imaging device provided in a mobile body operable by a user.

[0102] According to such a configuration, when the user operates the mobile body, the user can recognize a change in complexity that is difficult to notice from the target image as a change in the surface state of the three-dimensional object, so that the user can be prompted to pay attention to a slight change in the situation around the mobile body.

[0103] (12) In the information processing system described in (11) above, the moving body further includes a display unit and a perception information output unit other than the display unit. The display unit is configured to be able to display the three-dimensional object, and the perception information output unit is configured to output perception information that can be perceived by the user other than visually. Further, in the output step, the perception information is output from the perception information output unit in different manners according to the complexity parameter.

[0104] According to such a configuration, since the user can further perceive the complexity of the change in the environment around the moving body other than visually, it becomes easier to notice the change in the surrounding complexity.

[0105] (13) An information processing method including each step of the information processing system described in any one of (1) to (12) above.

[0106] (14) An information processing program for causing at least one computer to execute each step of the information processing system described in any one of (1) to (12) above.

[0107] (15) A mobile body, configured to be operable by a user, comprising an imaging device, a display unit, and at least one processor, wherein the imaging device is configured to generate a target image by imaging at least a part of the surroundings of the mobile body, the display unit is configured to display information visible to the user, and the at least one processor is configured to execute a program such that the following steps are performed: In a target image acquisition step, the target image is acquired; in a calculation step, a complexity parameter indicating the complexity of the temporal change of the target image is calculated based on the target image; and in a display control step, based on the complexity parameter, as the change per unit time of the target image becomes more complex, a three-dimensional object is displayed on the display unit such that the surface state of the three-dimensional object is spatially finely distorted compared to a reference state, and the reference state is the surface state of the three-dimensional object corresponding to the case where the complexity parameter is a reference value.

[0108] According to such a configuration, for a user operating the mobile body, a change in the complexity of the surroundings of the mobile body, which is difficult to notice from the target image, can be visually recognized in the form of a change in the surface state of the three-dimensional object, so that attention can be drawn to a change in the surrounding situation for the user operating the mobile body. Of course, this is not the limit.

[0109] Finally, although various embodiments according to the present disclosure have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and their modifications are included in the scope and gist of the invention and are also included in the invention described in the claims and its equivalent scope.

Explanation of Reference Numerals

[0110] 1: Information processing system 2: Vehicle 3: Sheet 31: Air Jack 4: In-vehicle Camera 5: Display Unit 6: Car Audio 7: Information Processing Device 70: Communication Bus 71: Communication Unit 72: Memory Unit 73: Processor 74: HMI Device IM0: Environmental Image IM0a: Still Image IM0b: Dynamic Image IM1: In-vehicle Image IM1a: In-vehicle Image IM1b: In-vehicle Image IM1c: In-vehicle Image IM1d: In-vehicle Image IM1e: In-vehicle Image IM1f: In-vehicle Image IM2: Modulated Image IM2a: Modulated Image IM2b: Modulated Image IM2c: Modulated Image IM2d: Modulated Image IM2e: Modulated Image IM2f: Modulated Image IM3: Display Image Ob1a: Static Object Ob1b: Movable Object Ob2: Three-dimensional Object R1a: Background Area R2: External Area ST0: Reference State U: Occupant

Claims

1. An information processing system, comprising at least one processor, wherein the at least one processor is configured to execute a program such that the following steps are performed: In a target image acquisition step, a target image that changes over time is acquired; In a calculation step, based on the target image, a complexity parameter indicating the complexity of the temporal change of the target image is calculated; In a display control step, based on the complexity parameter, as the change per unit time of the target image becomes more complex, the three-dimensional object is displayed such that the surface state of a predetermined three-dimensional object is spatially finely distorted compared to a reference state; wherein the reference state is the surface state of the three-dimensional object corresponding to the case where the complexity parameter is a reference value.

2. The information processing system according to claim 1, wherein in the display control step, the three-dimensional object is displayed such that the surface state is distorted by changing the three-dimensional effect of a predetermined environmental image displayed on the surface of the three-dimensional object.

3. The information processing system according to claim 2, wherein the environmental image is an image defined by a periodic pattern.

4. The information processing system according to claim 2, wherein in an additional image processing step, further, based on the complexity parameter, image processing is performed to change the environmental image, and in the display control step, based on the complexity parameter, the three-dimensional object is displayed by distorting the environmental image on which the image processing has been performed and which is displayed on the surface of the three-dimensional object.

5. The information processing system according to claim 2, wherein the environmental image is a video that changes over time.

6. The information processing system according to claim 1, wherein in the display control step, the three-dimensional object is displayed such that the shape of the three-dimensional object is distorted according to the complexity parameter.

7. The information processing system according to claim 1, wherein in the display control step, based on the complexity parameter, the three-dimensional object is displayed such that the manner of distorting the surface state of the three-dimensional object is changed.

8. The information processing system according to claim 7, The manner of distorting the surface state of the three-dimensional object includes changing the three-dimensional effect of the environmental image displayed on the surface of the three-dimensional object and changing the shape of the three-dimensional object, and the system.

9. In the information processing system according to claim 1, In the calculation step, the complexity parameter is calculated based on the difference between the first target image, which is the latest image in the target image, and the second target image, which is an image in the target image that is a predetermined period before the first target image.

10. In the information processing system according to claim 9, The second target image is the image immediately before the first target image in the target image, In the calculation step, the amount of change from the second target image to the first target image is calculated as the complexity parameter.

11. In the information processing system according to claim 1, The target image is a video captured by an imaging device provided in a mobile body operable by a user.

12. In the information processing system according to claim 11, The mobile body further includes a display unit and a perception information output unit other than the display unit, The display unit is configured to be able to display the three-dimensional object, The perception information output unit is configured to output perception information that can be perceived by the user other than visually, Furthermore, in the output step, the perception information is output from the perception information output unit in different manners according to the complexity parameter.

13. An information processing method, The method includes each step of the information processing system according to any one of claims 1 to 12.

14. An information processing program, A program that causes at least one computer to execute each step of the information processing system according to any one of claims 1 to 12.

15. A mobile body, Configured to be operable by a user, Comprising an imaging device, a display unit, and at least one processor, The imaging device is configured to generate a target image by imaging at least a part of the surroundings of the mobile body, The display unit is configured to display information visible to the user, The at least one processor is configured to execute a program so that the following steps are performed, In the target image acquisition step, the target image is acquired. In the calculation step, based on the target image, a complexity parameter indicating the complexity of the temporal change of the target image is calculated. In the display control step, based on the complexity parameter, as the change per unit time of the target image becomes more complex, the three-dimensional object is displayed on the display unit such that the surface state of the predetermined three-dimensional object is spatially finely distorted compared to the reference state. The reference state is the surface state of the three-dimensional object corresponding to the case where the complexity parameter is a reference value, the moving body.

Images