Image reading system, device and method, and response output system

Abstract

Provided is an image reading technology that can be used more appropriately. An image reading system includes a device that is carried or worn by a user, and reads an object using sound from an image captured by a camera, wherein the device automatically repeats capturing an image from a video of the camera at predetermined timing (step S2), analyzes the captured image to obtain information including text indicating an object within the image (step S3), determines an object and text that become a reading target by predetermined determination based on the obtained information (step S4), and automatically repeats sound reading of the text indicating the determined object from the device at predetermined timing (step S5).

Description

Technical Field

[0001] This disclosure relates to techniques for converting images / videos into text / strings, etc., and then using voice to read them aloud. Background Technology

[0002] There are technologies that use AI (artificial intelligence) to identify / analyze images / videos (including still images, videos, etc.) captured by cameras, convert them into text / strings (in other words, sentence-based conversion), and then read them aloud using voice (sometimes described as image-to-speech or voice-to-speech). As such a technology, for example, the "Seeing AI" app for Microsoft's iPhone, as shown in Non-Patent Document 1, is provided. In the "Seeing AI" app, photographic images obtained by capturing images with a camera are analyzed by AI, converted into text, and then read aloud using voice. Such technology is effective, for example, for assisting visually impaired individuals.

[0003] In addition, as examples of existing technology, there are Japanese Patent Application Publication No. 2000-325389 (Patent Document 1), Japanese Patent Application Publication No. 2006-251596 (Patent Document 2), and Japanese Patent Application Publication No. 2017-77446 (Patent Document 3), etc.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2000-325389

[0007] Patent Document 2: Japanese Patent Application Publication No. 2006-251596

[0008] Patent Document 3: Japanese Patent Application Publication No. 2017-77446

[0009] Non-patent literature

[0010] Non-patent literature 1:<URL:https: / / www.microsoft.com / ja-jp / ai / seeing-ai> "Seeing AI" Summary of the Invention

[0011] For example, in the aforementioned "Seeing AI" app, images captured by the user are transformed into text. Additionally, the "Seeing AI" app includes a function to take photos as "scenes" and describe the captured scene.

[0012] However, this technology does not adequately consider how to use it more appropriately when the user is walking, leaving room for improvement. For example, the workload of user-input operations, such as capture, is significant. As the surrounding environment changes over time while the user is walking, the user must input multiple times to keep track of the surroundings through audio narration of the camera image. Furthermore, while the user is walking, there are often various objects in front, behind, to the left, and to the right, creating a complex environment. It is difficult to handle such a complex situation with a single camera by simply reading audio from the camera image.

[0013] In prior art examples such as Patent Document 1, for example, a camera on a smartphone is described that reads images aloud based on a user's captured actions. However, in these prior art examples, specific implementation methods are not shown, such as how to best assist visually impaired individuals using images captured by the camera. Furthermore, these prior art examples do not consider preferred implementation methods when the camera is worn by a person. Moreover, in these prior art examples, appropriate functions, processing, and user convenience for scenarios such as assisting visually impaired individuals are not adequately considered.

[0014] The purpose of this disclosure is to provide a more appropriate technique for the aforementioned image reading technology.

[0015] Representative embodiments of this disclosure have the following structure. One embodiment provides an image reading system comprising a device carried or worn by a user, which reads aloud objects from images captured by a camera using sound. The device automatically and repeatedly captures images from the camera's footage at predetermined time intervals, analyzes the captured images to obtain information including text representing objects within the images, and, based on the obtained information, determines, through predetermined judgment, the object to be read aloud and the text, and automatically and repeatedly reads aloud the text representing the determined object from the device at predetermined time intervals.

[0016] According to representative embodiments of this disclosure, techniques that can be used more appropriately with respect to the above-described image reading technology can be provided. In specific embodiments, issues, structures, and effects other than those described above are shown. Attached Figure Description

[0017] Figure 1 This is a diagram showing the structure of the image reading system according to Embodiment 1.

[0018] Figure 2 This is a diagram illustrating other structures of the image reading system of Embodiment 1.

[0019] Figure 3This is a diagram illustrating other structures of the image reading system of Embodiment 1.

[0020] Figure 4 This is a diagram illustrating the processing flow of the image reading system and method according to Embodiment 1.

[0021] Figure 5A This is a diagram showing a structural example of the image reading device according to Embodiment 1.

[0022] Figure 5B This is a diagram showing a structural example of the image reading device according to Embodiment 1.

[0023] Figure 6A This is a diagram illustrating an example of using the image reading device according to Embodiment 1.

[0024] Figure 6B This is a diagram illustrating an example of using the image reading device according to Embodiment 1.

[0025] Figure 7A This is a diagram showing a structural example of the image reading device (equipment) in Embodiment 1.

[0026] Figure 7B This is a diagram illustrating a structural example of the server device in Embodiment 1.

[0027] Figure 8 This is a diagram illustrating a structural example of a predetermined timing for image capture and voice reading in Embodiment 1.

[0028] Figure 9A This is a diagram illustrating a structural example of multiple cameras in Embodiment 1.

[0029] Figure 9B This is a diagram illustrating an example of the ON / OFF operation of multiple cameras in Implementation 1.

[0030] Figure 9C This is a diagram illustrating a control example of multiple cameras in Embodiment 1.

[0031] Figure 10 This is a diagram showing an example of an object in Embodiment 1.

[0032] Figure 11 This is a diagram illustrating an example of the relationship between the user and the object in Implementation 1.

[0033] Figure 12 This is a supplementary explanatory diagram showing the priority of the moving direction in Embodiment 1.

[0034] Figure 13 This is a diagram illustrating the control mode in Implementation Method 1.

[0035] Figure 14 This is a diagram showing an example of a setting information table in Implementation Method 1.

[0036] Figure 15A This is a diagram illustrating an example of image recognition and voice reading in Implementation 1.

[0037] Figure 15B This is a diagram illustrating an example of image recognition and voice reading in Implementation 1.

[0038] Figure 16A This is a diagram illustrating an example of image recognition and voice reading in Implementation 1.

[0039] Figure 16B This is a diagram illustrating an example of image recognition and voice reading in Implementation 1.

[0040] Figure 17 This is a diagram illustrating an example of the processing sequence of the device and server apparatus in Embodiment 1.

[0041] Figure 18 This is a diagram illustrating the basic control related to periodic control in Implementation 1.

[0042] Figure 19 This diagram illustrates the control in Implementation 1 where voice reading is not performed without any changes.

[0043] Figure 20 This is a diagram illustrating the control of voice reading without change in the absence of any change in meaning in Implementation 1.

[0044] Figure 21 This is a diagram illustrating a control example of the reread instruction in Implementation 1.

[0045] Figure 22 This is a diagram illustrating examples of voice reading under different performances in Implementation 1.

[0046] Figure 23 This is a diagram illustrating a control example of the front and rear cameras in Embodiment 1.

[0047] Figure 24 This is a diagram illustrating a control example in Implementation 1 where the threshold used for judgment is different before and after.

[0048] Figure 25A This is a diagram illustrating an example of assistance for visually impaired individuals in Implementation 1.

[0049] Figure 25B This is a diagram illustrating an example of nighttime road assistance in Implementation 1.

[0050] Figure 25CThis is a diagram illustrating an example of the "using a mobile phone while walking" response in Implementation 1.

[0051] Figure 25D This is a diagram illustrating an example of vehicle assistance in Implementation 1.

[0052] Figure 26 This is a diagram illustrating a control example in Implementation 1 corresponding to the ability / inability to communicate with the server.

[0053] Figure 27 This is a diagram illustrating an example of voice reading corresponding to the priority of the object in Implementation 1.

[0054] Figure 28 This is a diagram illustrating an example of hazardous factors and safe space guidance in Implementation 1.

[0055] Figure 29 This is a diagram illustrating an example of voice reading that prioritizes a specific object in Implementation 1.

[0056] Figure 30 This is a diagram showing an example of an image from a 360-degree camera in Embodiment 1.

[0057] Figure 31 This is a diagram illustrating a control example of different sound output schemes based on the camera's shooting direction in Embodiment 1.

[0058] Figure 32 This is a diagram illustrating a control example of three-dimensional sound output in Embodiment 1.

[0059] Figure 33 This is a diagram showing the structure of the system in Embodiment 2.

[0060] Figure 34A This is a diagram showing the structure of the device in Embodiment 2.

[0061] Figure 34B This is a diagram showing the structure of the server in Embodiment 2.

[0062] Figure 35 This is a diagram illustrating the processing flow in Implementation Method 2.

[0063] Figure 36A This is a diagram illustrating a structural example of the system in Embodiment 2.

[0064] Figure 36B This is a diagram illustrating a structural example of the system in Embodiment 2.

[0065] Figure 36C This is a diagram illustrating a structural example of the system in Embodiment 2.

[0066] Figure 36D This is a diagram illustrating a structural example of the system in Embodiment 2.

[0067] Figure 37 This is a diagram showing a more detailed structural example of the system in Embodiment 2.

[0068] Figure 38 This is a diagram illustrating an example of the structure of the request data in Implementation 2.

[0069] Figure 39 This is a diagram illustrating an example of the surrounding conditions in Embodiment 2.

[0070] Figure 40 This is a diagram illustrating an example of multiple images in Embodiment 2.

[0071] Figure 41 This is a diagram showing the surrounding conditions in the first specific example of Embodiment 2.

[0072] Figure 42 This is a diagram illustrating one of the session flows in the first specific example of Embodiment 2.

[0073] Figure 43 This is a diagram illustrating the second flow of the session in the first specific example of Embodiment 2.

[0074] Figure 44 This is a diagram illustrating the flow of a session in the second specific example of Implementation 2.

[0075] Figure 45 This is a diagram showing an example of the screen display of the device in Embodiment 2. Detailed Implementation

[0076] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are generally used for the same parts, and repeated descriptions are omitted. In the drawings, the actual position, size, shape, and extent of the constituent elements are sometimes not shown for ease of understanding of the invention.

[0077] In descriptions of program-based processing, the main focus may be on programs, functions, and processing units. However, the core hardware component is the processor, or a controller, device, computer, or system comprised of such a processor. The computer, through the processor, executes processing according to a program read from memory while appropriately utilizing resources such as memory and communication interfaces. This achieves the intended functions and processing units. Processors may be, for example, semiconductor devices such as CPUs / MPUs and GPUs. Processing is not limited to software program processing; even dedicated circuits can be installed. Dedicated circuits can utilize FPGAs, ASICs, CPLDs, etc.

[0078] The program can be pre-installed as data on the target computer, or it can be distributed as data from a program source to the target computer. The program source can be a program distribution server on a communication network, or a non-transitory computer-readable storage medium such as a memory card or disk. The program can also consist of multiple modules. The computer system can also consist of multiple devices. The computer system can also consist of client / server systems, cloud computing systems, IoT systems, etc. Various data and information can be constructed using tables, lists, etc., but are not limited to these. The representation of identification information, identifiers, IDs, names, numbers, etc., can be interchanged.

[0079] <Implementation Method 1>

[0080] use Figure 1 The following figures illustrate the image reading system of Implementation Method 1, etc.

[0081] In the image reading system and method of Embodiment 1, a device equipped with a camera and an audio output device (also described as an image reading device) is used as a device carried or worn by the user. This device is equipped with at least a camera (in other words, a shooting function) and an audio output device (in other words, an audio output function), as well as other necessary software / hardware. This device includes wearable devices, such as smartphones, smartwatches, tablet terminals, smart glasses, head-up displays (HMDs), shoulder-worn devices, etc. Alternatively, the device can be a hat, clothing, bag, shoe, etc., equipped with a camera. The device can also be attached to a hat, clothing, etc.

[0082] The image reading system in Implementation 1 utilizes multiple cameras mounted on a device that serves as an image reading apparatus. These multiple cameras can be, for example, cameras positioned in two directions (front and back) or cameras positioned in four directions (front, back, left, and right). Each camera can also be a sensor with features such as stereo imaging or rangefinding capabilities. The number and orientation of the cameras mounted on the device are not limited and can be arranged in various ways.

[0083] The device captures (in other words, photographs) images from the input images of each camera. This image (also referred to as a captured image) is a photographic image obtained by photographing the surrounding environment relative to the user and the device, in a predetermined direction, i.e., the camera's shooting direction (e.g., the front-to-back direction). In this image, various objects such as pedestrians and vehicles are captured as part of the surrounding environment.

[0084] The device can automatically capture images from the camera's input footage at predetermined time intervals (periodically, designated as the first time interval). This results in a series of images capturing the user's surroundings continuously or intermittently over time. The image reading system detects objects in the surrounding environment from the image series through analysis, including recognition using AI (e.g., machine learning models). For example, the image reading system extracts features corresponding to objects from the image series using AI recognition, detecting objects based on features that vary significantly over time between images.

[0085] The image-to-speech system determines whether to include a detected object in the audio reading process based on predetermined criteria. These criteria might include, for example, the distance between the object and the user / device. Furthermore, when multiple objects are present, the system also determines the priority of the audio readings, or in other words, the output order.

[0086] Image-to-speech systems transform objects identified by AI into text representing those objects. As a result of this identification, the AI ​​(e.g., a machine learning model) responds with information such as the text representing the object, the location of features corresponding to the object, and its size.

[0087] Regarding the objects (such as pedestrians) that are read aloud, the image-to-speech system uses the device to read aloud the text representing the objects in a priority order and outputs the sound from the sound output device. The text and sound read aloud are, for example, "pedestrian" or "There is a pedestrian ahead," etc.

[0088] The image-to-speech system also automatically reads the object aloud at predetermined time intervals (periodically, designated as the second time interval). The audio reading period (second time interval) can also differ from the image capture period (first time interval).

[0089] As described above, the device, which functions as an image reading device, automatically and repeatedly captures images continuously or intermittently at predetermined time intervals, correspondingly performs text transformation using AI recognition, and automatically and continuously reads aloud based on the captured images at predetermined time intervals. This image capture and audio reading process requires virtually no user input, such as capture instructions.

[0090] The image-to-speech system stores the historical data / information of the aforementioned continuous image capture and audio reading in storage resources, for example, for a predetermined period of time, so that it can be referenced later.

[0091] [System Structure (1)]

[0092] Figure 1 The structure of the image reading system of Embodiment 1 is shown. Figure 1 The image reading system is a system that includes device 1 as an image reading device. Figure 1 The system has a device 1 that is appropriately connected via communication network 9 and a server device 2 on a cloud computing system 3.

[0093] In other words, device 1 is a user terminal (portable terminal, installation terminal, information processing terminal, computer, etc.) carried or worn by user U1. User U1 can be either a visually impaired person or a non-visually impaired person. As specific examples, device 1 can include a group of wearable devices such as smartphone 1A, smartwatch 1B, smart glasses (or HMD) 1C, hat 1D, clothing 1E, and shoulder-worn device 1F.

[0094] In addition, a single user U1 can use multiple devices 1. The multiple devices 1 can also collaborate with each other through communication.

[0095] Device 1 includes control function 10, communication function 11, shooting function 12, and sound output function 13. Control function 10 performs control related to the image reading function. Communication function 11 communicates with server device 2 (which serves as AI server 2) via communication network 9. Shooting function 12 captures images using a camera. Sound output function 13 outputs sound from a speaker, headphone jack, or other sound output device. Additionally, although not shown, device 1 may also include display function and sound input function. Device 1 at least includes shooting function 12 and sound output function 13. (The following text is incomplete and requires further context.) Figure 7A Provide details about device 1.

[0096] Server device 2 possesses the functionality of AI20, in other words, it possesses AI server 2. AI20, in other words, has parsing and recognition functions. This AI20 is software / hardware that identifies objects (in other words, features) from input images, transforms them into text representing those objects (features), and outputs information about the objects, including the text. This information is then used... Figure 7B This section describes the details of server device 2.

[0097] Device 1 and server device 2 can also function as a client / server system. Device 1 sends requests (e.g., images) related to the recognition of AI 20 to server device 2 as needed. Server device 2 performs recognition using AI 20 based on the request and sends the recognition results (e.g., text) back to device 1 as a response.

[0098] In this embodiment, the AI20 of server device 2 performs text-to-speech conversion. The control function 10 of device 1 performs text-to-speech conversion. Figure 4 (Step S5 in the process). Not limited to this, as a variation, the AI20 of the server device 2 can also perform a transformation from text to sound after the transformation to text, and output the transformed sound data to the device 1.

[0099] Furthermore, in the embodiments described later, it is possible to switch between identification using the AI20 of the server device 2 and identification within the device 1, depending on factors such as the communication status between the device 1 and the server device 2. In this case, as a corresponding component, it is also possible to... Figure 1 Device 1 has an AI function 20b. This AI function 20b is a copy of the function of AI 20 of server device 2, or a simplified analysis / recognition function with lower performance than AI 20.

[0100] [System Structure (2)]

[0101] Figure 2 The system structure of the variant example is shown. Figure 2 The system connects a device 1a (in other words, an input / output device) with a shooting function 12 and a sound output function 13, a device 1b with a control function 10 and a communication function 11, and a server device 2 that acts as an AI server 2. Device 1a can be, for example, smart glasses 1C. Device 1b can be, for example, a smartphone 1A. The system combines device 1a and device 1b with... Figure 1 The equipment is equivalent to 1.

[0102] [System Structure (3)]

[0103] Figure 3 The system structure of the variant example is shown. Figure 3 The system is also integrated and installed in a single device such as a smartphone 1A. Figure 1 The system has the AI20 function of AI server 2. In this case, since device 1 has AI20 (in other words, analysis function, recognition function), it is basically not necessary to access server device 2 of communication network 9 when recognizing camera images. Figure 1 The AI20 within device 1 can be downloaded and installed from the operator's server device on communication network 9, and can also be updated and upgraded via communication as needed.

[0104] [AI]

[0105] Figure 1AI Server 2's AI20, in other words, is a large-scale database (DB) for image parsing and text transformation. As AI20, it can apply a multimodal large language model (MLLM). MLLM is an LLM capable of processing multiple types of data / information such as images, text, and sound. In this embodiment, the MLLM applied to AI20 has the function of learning from image groups to describe / represent the shape, position, etc., of characteristic parts / regions / objects within an image using words / text / strings / natural language.

[0106] As a technology applicable to AI20 MLLM, it is possible to apply known technologies, such as those shown in References 1 and 2 below.

[0107] [Reference Information 1]

[0108] <https: / / arxiv.org / abs / 2306.13549> , "A Survey on Multimodal LargeLanguage Models"

[0109] [Reference Information 2]

[0110] <https: / / browse.arxiv.org / pdf / 2306.13549.pdf> , "A Survey on MultimodalLarge Language Models"

[0111] [Usage Scenarios]

[0112] The image-to-speech system of Implementation Method 1 can be widely used as an aid to assist users while walking. This system can be used in various scenarios. As more specific examples, the following scenarios can be cited.

[0113] 1. Assistance for the visually impaired: In this application, objects in the user's surroundings are detected and read aloud to assist the visual perception of the user, who is visually impaired. (Further details to follow.) Figure 25A The text illustrates an example of assisting visually impaired individuals. For instance, in assisting visually impaired individuals with normal hearing, it is difficult or impossible for them to visually perceive their surroundings while walking. However, an image-to-speech system transcribes the surrounding situation from camera images and reads it aloud. Thus, visually impaired individuals can use their hearing to identify objects in their surroundings, contributing to traffic safety and convenience.

[0114] 2. Nighttime Road Assistance: In this application, for example, in dimly lit conditions such as nighttime roads or areas with poor security, it detects suspicious individuals approaching the user from behind and provides audio alerts to alert them. (More details to follow.) Figure 25B The image shows an example of night driving assistance.

[0115] 3. "Walking with a mobile phone" assistance: In this application, while ensuring the user's safety in their surroundings, the system assists in detecting and reading aloud surrounding objects while walking or stationary using a smartphone (e.g., 1A). This will be discussed later. Figure 25C The example shown illustrates how to assist with "using a mobile phone while walking." Furthermore, this application is not intended to promote the use of smartphones or other devices when safety in the user's surroundings is not guaranteed, such as "using a mobile phone while walking" or "using a mobile phone while driving," but rather to contribute to the safety of the user and their surroundings.

[0116] 4. Vehicle Driving Assistance: This application, for example, assists users when riding electric bicycles, electric scooters, or other similar vehicles by detecting surrounding objects and providing audio guidance. This application is not intended to promote the use of smartphones or similar devices while driving, which could potentially cause accidents, but rather to contribute to the safety of the user and those around them. (Further details will follow.) Figure 25D The image shows an example of vehicle driving assistance.

[0117] It is effective even for users who are not visually impaired, as mentioned above, such as night driving assistance, "walking while using a mobile phone" coping assistance, and vehicle driving assistance.

[0118] [Image reading methods and processing flow]

[0119] Figure 4 The processing flow of the image reading method as an embodiment 1 is shown. Figure 1 System execution Figure 4The processing flow is as follows: In step S1, user U1's device 1 turns on the image reading function. In step S2, device 1 captures an image from the camera at a predetermined time. In other words, device 1 takes a picture of the image. In step S3, the system analyzes the captured image obtained in step S2. The analysis includes object identification using AI20. In this system, as a result of the analysis, information including text representing the object (in other words, words, strings, sentences, descriptions, etc.) is obtained. In step S4, based on the result of step S3, the system determines the object and text to be read aloud through a predetermined judgment (e.g., a judgment of the degree of change described later). In step S5, the system reads aloud the text representing the object determined in step S4 from device 1 at a predetermined time (in other words, outputs sound). In step S6, device 1 checks whether the image reading function is turned off. If it is off ("Yes"), the process ends; if it remains on ("No"), it returns to step S1, and the process is repeated.

[0120] [Equipment Example]

[0121] exist Figure 5A In the middle, as Figure 1 The example of device 1 shows an installation example of cameras, etc., in smart glasses 1C. In this example, the smart glasses 1C are equipped with multiple cameras that use four directions—front, back, left, and right—as shooting directions. Furthermore, Figure 2 The coordinate system WU in the diagram represents a coordinate system (X, Y, Z) with user U1 and device 1 as references. When viewed from user U1, the X-axis / X direction is the left-right direction, the Y-axis / Y direction is the front-back direction, and the Z-axis / Z direction is the up-down direction.

[0122] The Smart Glasses 1C features cameras C11 and C12 positioned to the left and right on the front, shooting forward; and cameras C21 and C22 positioned to the left and right on the back, shooting backward. Additionally, the Smart Glasses 1C includes cameras C31 and C32 positioned to the right on the right side (front and back), shooting to the right; and cameras C41 and C42 positioned to the left on the left side (front and back), shooting to the left. Cameras C11 and C12 shoot forward (+Y direction), meaning they are front cameras. Cameras C21 and C22 shoot backward (-Y direction), meaning they are rear cameras. Cameras C31 and C32 shoot to the right (+X direction), meaning they are right cameras. Cameras C41 and C42 shoot to the left (-X direction), meaning they are left cameras.

[0123] In this example, the front camera functions as a stereo camera (in other words, a sensor with ranging capabilities) implemented using two cameras C11 and C12, but it is not limited to this. A single camera may also be used as the front camera, etc. Regarding the determination of relative distances, which will be described later, this can be done by analyzing the image using AI20, but the ranging function of such a stereo camera, etc., can also be utilized.

[0124] The remote control 1c can also be attached to the smart glasses 1C or other devices. Alternatively, other smartphones such as smartphones 1A can also function as remote controls for the smart glasses 1C. When the user U1 inputs an operation on a button or other device on the remote control 1c, the remote control 1c sends an instruction signal to the smart glasses 1C. The smart glasses 1C then performs an action according to the signal.

[0125] In addition, although not shown, the smart glasses 1C also include a display, microphone, speaker, vibrator, and battery. The sound output device is not limited to a speaker; bone conduction headphones or similar devices can also be used.

[0126] Furthermore, even if user U1 is not visually impaired, they can still observe image information displayed on the display surface 1Cd of the smart glasses 1C.

[0127] exist Figure 5B In the example shown, as another example of device 1, a shoulder-worn device 1F is shown. The shoulder-worn device 1F is a type of device that is worn on the shoulder or head. Figure 5B The shoulder-mounted device 1F has two rod-shaped frames on the left and right sides and a semi-circular frame connecting the back side of these frames. Multiple cameras capable of shooting in four directions (front, back, left, and right) are mounted on the shoulder-mounted device 1F. These multiple cameras are... Figure 5A The multiple cameras are identical.

[0128] The above example illustrates a setup with cameras in four directions: front, back, left, and right. However, this is not a limitation; any camera capable of capturing images in at least one direction can be used. Alternatively, the left and right cameras (C31, C32, C41, C42) can be omitted, and only the front and back cameras (C11, C12, C21, C22) can be installed. With cameras capturing images of the rear (C21, C22), the situation behind the user can be appropriately captured. With cameras capturing images of the left and right directions (C31, C32, C41, C42), the situation in the left and right directions can be appropriately captured for the user. Furthermore, for example, if the front camera has a sufficiently large field of view (e.g., close to 180 degrees), the situation in the left and right directions can also be captured for the user.

[0129] [Device usage method: Smartphone]

[0130] Figure 6A This illustrates an example of how the device 1 is a smartphone 1A. Figure 6A The image shows user U1 holding and using smartphone 1A, with the camera on the back (front and rear) of smartphone 1A taking pictures of user U1 in the front-to-back direction (±Y direction). Figure 6A In the diagram, user U1 holds smartphone 1A with the frame nearly vertical. Camera C1 is a front camera (external camera) positioned on the back of the flat frame of smartphone 1A (the side without a display screen), with the optical axis of the shot facing forward (+Y). Camera C2 is a rear camera (internal camera) positioned on the surface of the flat frame of smartphone 1A (the side with the display screen), with the optical axis of the shot facing backward (-Y).

[0131] In addition, users U1 who are not visually impaired can also observe and utilize the image information displayed on the screen 1Ad of their smartphone 1A as needed.

[0132] Furthermore, the user U1 can hold the smartphone 1A in a different direction than the front shown in the diagram. In this case, the front camera C1 can capture images in that direction. Additionally, if the smartphone 1A is equipped with a camera that can capture images in other directions, that camera can also be used.

[0133] [Equipment usage: hats, clothes]

[0134] exist Figure 6B In this example, as another way of utilizing device 1, cameras are shown mounted on a hat 1D and clothing 1E to capture images of the user U1 from the front and back. For example, the hat 1D has a camera C3 mounted on its front surface to capture images from the front (+Y) and a camera C4 mounted on its back to capture images from the back (-Y). Alternatively, clothing 1E has a camera C5 mounted on its front to capture images from the front (+Y) and a rear camera C6 mounted on its back to capture images from the back (-Y). These cameras can also be connected to the main device 1 (e.g., a smartphone 1A) via communication. Similarly, left and right cameras can also be mounted.

[0135] Furthermore, device 1 can also be installed on the user U1's body, clothing, bag, shoes, etc. For example, a camera or device 1 with a camera can be installed in the pocket / hanger of clothing 1E. The user U1 can also wear / carry multiple devices 1 such as smartphones 1A. Even in situations like... Figure 6B In this case, device 1 can also use front and rear cameras to capture the forward and backward direction of user U1.

[0136] [Example of equipment structure]

[0137] Figure 7A An example of the structure of device 1 is shown. Device 1 includes a processor 301, a memory 302, a non-volatile memory (in other words, a storage device) 303, a camera (in other words, a shooting unit) 304, an image processing unit 305, an image storage unit 306, a position detection unit 311, a posture detection unit 312, a bio-information detection unit 313, a button / operation unit 314, a display / show unit 315, a microphone / sound input unit 316, a speaker / sound output unit 317, a connector / input / output interface 318, a battery / power supply unit 319, a vibration unit 320, and a communication unit 330. The communication unit 330 is equipped with a communication interface, for example, including a first wireless communication unit 331, a second wireless communication unit 332, a third wireless communication unit 333, and a wired communication unit 334. Each wireless communication unit has its own wireless communication interface, for example, an interface corresponding to a mobile network, wireless LAN, Bluetooth (registered trademark) etc. In the camera 304, multiple cameras are included, for example, cameras #1 to #8. These cameras, for example, are with Figure 5A There are 8 cameras in front, behind, left, and right.

[0138] The processor 301, consisting of a CPU and other components, controls the entire device 1 and its individual parts. The memory 302 stores the data / information processed by the processor 301. The processor 301 reads programs from the non-volatile memory 303 into the memory 302 and executes the processing according to the programs. Thus, various functions are implemented as execution modules.

[0139] The non-volatile memory 303 stores various data / information, such as programs like OS 341, image reading software 342, AI data 343, user information 344, setting information 345, and access destination information 346. The image reading software 342 is data such as programs used to implement the image reading function in Embodiment 1. The AI ​​data 343 is data such as recognition results exchanged with the AI ​​20 of the server device 2. Furthermore, when recognition processing is performed on the device 1 side, the AI ​​data 343 includes data corresponding to the AI ​​20. The user information 344 is information about the user U1. The setting information 345 includes system setting information and user setting information. The access destination information 346 is information used for communication with the server device 2, etc.

[0140] The camera 304 includes circuitry for controlling shooting using multiple cameras. The image processing unit 305 processes the images captured by the camera 304. The image storage unit 306 stores data such as the images captured by the camera 304 and the images processed by the image processing unit 305.

[0141] The position detection unit 311 uses, for example, GNSS to detect the position of device 1 and obtain position information. The posture detection unit 312 includes a posture sensor and detects the posture of device 1. The biometric information detection unit 313 detects the fingerprint of user U1 for fingerprint authentication, or detects the face of user U1 for facial authentication. The button / operation unit 314 includes a power button, volume buttons, etc., and is subject to operation inputs from user U1. The display / shown unit 315 processes the display to show images / pictures. The microphone / sound input unit 316 inputs / recognizes sound via a microphone. The speaker / sound output unit 317 outputs sound via a speaker, headphone jack, etc. The connector / input / output interface 318 is equipped with an input / output interface for connecting input / output devices. The battery / power supply unit 319 supplies necessary power to each component based on a battery. The vibration unit 320 vibrates as needed.

[0142] [Example of server structure]

[0143] Figure 7B An example of the structure of server 2 is shown. Server 2 includes a processor 401, a memory 402, a non-volatile memory (in other words, a storage device) 403, an operation input unit 414, a connector / input / output interface 418, a battery / power supply unit 419, and a communication unit 430. The communication unit 430 is equipped with communication interfaces, such as a first wireless communication unit 431, a second wireless communication unit 432, a third wireless communication unit 433, and a wired communication unit 434.

[0144] The processor 401, consisting of a CPU / GPU, controls the entire server device 2 and its individual components. The memory 402 stores the data / information processed by the processor 401. The processor 401 reads programs from the non-volatile memory 403 into the memory 402 and executes the processing according to the programs. Thus, it performs various functions as an execution module.

[0145] Various data / information are stored in the non-volatile memory 403, such as programs like OS 441, AI software 442, AI data 443, user information 444, setting information 445, and access destination information 446. The AI ​​software 442 is used to perform analytical processing, including image recognition. Figure 1 The AI20 program and other data. AI data 443 includes model and image data used for AI20's recognition processing, recognition results exchanged between AI20 and device 1, and other data. User information 444 is user U1's information. AI setting information 445 is AI20's setting information. Access destination information 446 is information used for communication with device 1, etc.

[0146] Operation input unit 414 accepts operation input. Connector / Input / Output Interface 418 is equipped with an input / output interface for connecting input / output devices. Battery / Power Supply Unit 419 supplies necessary power to each unit based on a battery.

[0147] [Pre-set timing for image capture, etc.]

[0148] Figure 8 An example is shown of the relationship between the input image 801 (time point, image frame) of the camera 304 of device 1, image capture 802, recognition 803 performed using AI 20, and the predetermined timing, period, etc. in the voice reading 804.

[0149] The input image 801 has image frames at various time points in a time sequence. It also has image frames at a predetermined frame period of the camera 304. For example, at time point 1, there is frame f1. Image capture 802 is performed automatically, continuously or intermittently, at a preset period (first time interval). Images are captured from the image frames of the input image 801 at this period. For example, frame 1 at time point 1 is captured as image 1, and frame 3 at time point 3 is captured as image 2. In this example, image capture 802 is performed at a period of every 2 frames. However, in practice, for example, it could be capture once every 8 frames. The captured images are saved to memory (image storage unit 306). Image capture can be either capturing still images from each camera or capturing a frame of video during video recording.

[0150] For each captured image, AI20 identification 803 is performed. In this example, AI20 identification 803 is performed at the same period as image capture 802. Regarding objects detected and determined as objects for audio reading based on the results of identification 803, audio reading 804 is automatically performed at a pre-set period (second time interval). For example, audio reading 1 is performed for an object detected from image 1 at time point 1, and audio reading 2 is performed for an object detected from image 3 at time point 3. In this example, audio reading 804 is performed at a period of 4 frames. Furthermore, if no object is detected based on the results of image identification 803, and the detected object is not considered as an object for audio reading, no audio reading 803 is performed even at the time corresponding to the period.

[0151] Device 1 stores the historical data (including captured images, recognition results, etc.) processed as described above, such as image capture 802, recognition 803, and voice reading 804, in storage resources (e.g., non-volatile memory 403) for at least a predetermined time. Device 1 may also send this historical data to a server device 2 or the like for storage.

[0152] Device 1 is controlled to prioritize audio reading of an object when the captured image at the current timer shows different activity or significant changes compared to the captured image at the previous timer. Specific examples will be described later.

[0153] [Number of cameras and shooting direction]

[0154] Figure 9A This is a schematic diagram, viewed from above in the vertical direction, showing an example of the number of cameras 304 on device 1 of user U1 and their shooting direction. State A is the first example. User U1 is walking, for example, on sidewalk 901. User U1 carries or wears device 1 (e.g., smartphone 1A). The first example is as follows... Figure 6A The two cameras (C1, C2) on the back of the smartphone 1A are used as an example of front and rear cameras. In the accompanying drawings, for ease of understanding, the front and rear cameras (C1, C2) are illustrated separately. DC1 is the shooting direction of the front camera (C1), which is the +Y direction corresponding to the front of the user U1. DC2 is the shooting direction of the rear camera (C2), which is the -Y direction corresponding to the rear of the user U1. In this example, both the front camera (C1) and the rear camera (C2) have wide-angle lenses, with a shooting range / field of view approaching 180 degrees. The semicircle is a schematic image illustrating the shooting range / field of view. Application Figure 6B The situation is the same.

[0155] State B is the second example. The second example uses, for example... Figure 5A Example of four cameras (front, back, left, and right) in a smart glasses 1C. User U1 wears device 1 (e.g., smart glasses 1C). Here, the front camera is designated C10, the rear camera C20, the right camera C30, and the left camera C40. DC10 is the shooting direction (+Y) of the front camera C10. DC20 is the shooting direction (-Y) of the rear camera C20. DC30 is the shooting direction (+X) of the right camera C30. DC40 is the shooting direction (-X) of the left camera C40. The dashed triangles are schematic images of the shooting range / field of view of each camera, with a field of view of at least 45 degrees and possibly close to 180 degrees. Application Figure 5B The situation is the same.

[0156] The camera 304 is not limited to a structure in which it is integrally mounted on the device 1, but can also be a structure in which the camera device is externally connected to the device 1 (e.g., smartphone 1A) as the main body. The camera device can also be worn on a predetermined position on the body (e.g., hat 1D, clothing 1E).

[0157] [Camera on / off]

[0158] In other embodiments, it can also be set and used in such a way that only a portion of the multiple cameras of device 1 are turned on. Figure 9B Examples of multiple cameras being turned on / off are shown. State A is an example where only the front camera C10 is turned on out of the four cameras in the front, back, left, and right directions of the smart glasses 1C. State B is an example where both the front camera C10 and the rear camera C20 are turned on. State C is an example where the front camera C10, the right camera C30, and the left camera C40 are turned on. State D is an example where both the right camera C30 and the left camera C40 are turned on.

[0159] [Example of controlling multiple cameras]

[0160] In addition, when using multiple cameras such as front and rear cameras of device 1, the processing from image capture from each camera to audio reading is performed simultaneously in parallel. Figure 8 The scenario of (etc.) can also be considered as an example. However, in this case, the processing load is high, and there are concerns about latency, etc. Therefore, as an example, the processing of images from each of the multiple cameras can also be controlled by dividing the time into predetermined intervals and frequencies, in other words, by performing time separation. For example, in the processing of capture, recognition, and sound reading from the front camera (C1) and the processing of capture, recognition, and sound reading from the rear camera (C2), the overall scheduling can also be controlled by dividing the time into predetermined intervals and frequencies. For example, the following control example can be given.

[0161] First, for processing tasks with low parallel processing load, such as image capture and image saving, it can be set up so that both the front and rear cameras perform the same repeated operations. When the parallel processing load for these image capture and image saving operations is high, they can be processed sequentially by dividing the time interval. Furthermore, when device 1 reads images from memory based on the captured images from each camera and performs subsequent processing, it divides the time interval on a per-camera image basis. For example, it can also process images from the front and rear cameras alternately.

[0162] Figure 9A Control 1 illustrates a control example for the front and rear cameras corresponding to state A. For example, the processing (recognition, voice reading) of the front camera (C1) image is performed at time points 1 and 3, and the processing (recognition, voice reading) of the rear camera (C2) image is performed at the next time points 2 and 4, alternating repeatedly. In this example, the timing of the voice reading during the processing of the front camera (C1) image (second time interval) and the timing of the voice reading during the processing of the rear camera (C2) image (second time interval) are alternating, with a processing frequency ratio of 1:1.

[0163] Figure 9A Control 2 illustrates an example of controlling the front, rear, left, and right cameras corresponding to state B. For instance, the processing of the front camera (C10) image at time point 1, the rear camera (C20) image at the next time point 2, the right camera (C30) image at the next time point 3, and the left camera (C40) image at the next time point 4 are performed sequentially. These four processing directions are grouped together, and this process is repeated. In this example, the audio reading period (the second time interval) during image processing in each direction is the same, with staggered timings, resulting in a processing frequency ratio of 1:1:1:1.

[0164] As another control example, it can be set to a state where only the images from the front camera (C1) are processed repeatedly under normal conditions, and then changed to a state where only the images from the rear camera (C2) are processed repeatedly under predetermined conditions. Alternatively, it can be set to a state where only the images from the front camera (C1) are processed repeatedly under normal conditions, and then changed to a state where the processing of images from the front camera (C1) and the processing of images from the rear camera (C2) are performed alternately under predetermined conditions.

[0165] Figure 9C Showing in with Figure 9A State A is a control example that changes the processing ratio of each camera when there is at least one front and rear camera (C1, C2) in device 1. Control A is an example of repeatedly processing the image of the front camera (C1). Control B is an example of processing the image of the front camera (C1) while primarily processing the image of the front camera (C1), and also inserting the processing of the image of the rear camera (C2) at a frequency of 3:1. Control C is an example of alternating the processing of the image of the front camera (C1) and the processing of the image of the rear camera (C2) at a frequency of 1:1. Figure 9A (Same as control 1). Control D is an example where image processing is mainly performed on the images of the rear camera (C2), while image processing of the front camera (C1) is also inserted at a frequency of 3:1. Control E is an example where image processing of the rear camera (C2) is repeated only.

[0166] The image-to-speech system can also switch between these multiple controls in a timely manner based on user input and pre-defined judgments.

[0167] From the perspective of computing power, it is sometimes difficult to achieve real-time and high-precision voice reading of all the surrounding conditions of the user in all directions. Furthermore, even if the surrounding conditions in all directions are read aloud simultaneously, there is still a possibility of user confusion. In these cases, based on the user and the usage scenario, the priority should be given to identifying / reading from which direction by setting / selecting, as mentioned above, the camera usage, processing ratio, control mode, etc.

[0168] For example, control A can be used when the user requires no rear-end assistance. Control E can be used when the user requires only rear-end assistance. Furthermore, it is possible to perform controls such as using control C normally for good balance in front-to-back positioning, switching to control B when focusing on an object in front, and switching to control D when focusing on an object behind (described later). Figure 23 ).

[0169] [User's surroundings]

[0170] Figure 10 This is an explanatory diagram relating to the surrounding environment when a user U1 wearing device 1 (e.g., smart glasses 1C) is stationary or walking on sidewalk 901, and is a schematic diagram viewed from above in the vertical direction. It has forward, backward, left, and right directions when viewed from the user U1's perspective. In this example, it is assumed that the user U1 is walking forward (+Y direction) on sidewalk 901. The white arrows indicate the direction of walking / movement. In this example, there is a bicycle lane 902 and a road 903 on the right side of sidewalk 901.

[0171] In the illustrated example, examples of objects that become the subjects of detection include a pedestrian 1001, a bicycle 1002, and a car 1003. Pedestrian 1001 is walking on sidewalk 901 in the -Y direction (as shown in the illustration), approaching the front of user U1. From user U1's perspective, pedestrian 1001 is in front. Bicycle 1002 is traveling on bike lane 902 in the -Y direction, approaching user U1 from the diagonal front. From user U1's perspective, bicycle 1002 is to the right diagonally forward. Car 1003 is traveling on lane 903 in the +Y direction. From user U1's perspective, car 1003 is on the right. These objects become the subjects of the audio reading.

[0172] [The positional relationship between the user and the object]

[0173] Figure 11 This example illustrates the positional relationship between user U1 and an object. It uses a pedestrian 1001 facing in front of user U1 as an example. Figure 10Let's illustrate with examples. State A is when user U1 is stationary and pedestrian 1001 is stationary. State B is when user U1 is stationary and pedestrian 1001 is walking. State C is when user U1 is walking and pedestrian 1001 is stationary. State D is when user U1 is walking and pedestrian 1001 is walking.

[0174] Regardless of the state, the relative positional relationship and speed between user U1 and pedestrian 1001 can be considered. Furthermore, the dashed circle represents a predetermined distance range 1100 centered on the positions of user U1 and device 1. The radius 1101 of the circle within range 1100 is approximately equivalent to a threshold for the relative distance between user U1 and the object. Figure 11 In the example, the threshold for relative distance is the same in all directions. In states A through D, the relative distance DA between user U1 and pedestrian 1001 is given. In state A, the relative speed between user U1 and pedestrian 1001 is 0. In state B, with pedestrian 1001's speed set to v2 (-v2), the relative speed is v2 (-v2). In state C, with user U1's speed set to v1 (+v1), the relative speed is v1 (+v1). In state D, with pedestrian 1001's speed set to v2 (-v2) and user U1's speed set to v1 (+v1), the relative speed is +v1 - (-v2) = v1 + v2.

[0175] Regarding the relative distance and relative speed mentioned above, in this embodiment, they can be determined primarily based on the analysis of the camera's image. Furthermore, it is not limited to this; various sensor functions installed on device 1 can also be used to determine the position, orientation, and speed of user U1, the position, orientation, and speed of object, and the relative distance and speed between user U1 and object.

[0176] Furthermore, the relative distance and speed described above can be used for predetermined control. For example, when the relative distance between user U1 and the object is within a threshold, it can be used as the object for audio reading or as the object for warning. For example, when the relative distance DA is within a predetermined distance range 1100, it can also be used as the object for audio reading. Additionally, the predetermined distance range 1100 (and its corresponding threshold) can be set in multiple stages. For example, when the relative distance DA is within a first range, it can be used as the object for audio reading, and further, when it is within a smaller second range, it can be used as the object for audio reading with an added warning. Similarly, when the relative speed between user U1 and the object is above a threshold, it can be used as the object for audio reading or as the object for warning.

[0177] Furthermore, in this embodiment, the relative relationship between the user and the objects is determined by processing the amount of change in objects between captured images (described later). Therefore, regarding such... Figure 11 The various situations corresponding to combinations such as the user being stationary or moving, and the object being stationary or moving, can be handled in the same way.

[0178] [The relationship between the user's movement direction and the camera's direction]

[0179] In addition, as a supplement, Figure 12 This diagram illustrates the relationship between the movement direction of user U1 and the shooting direction of camera 403. The movement direction of user U1 may be consistent with or inconsistent with the camera's shooting direction.

[0180] State A is the case where user U1's device 1 has front and rear cameras. User U1 moves forward (in the +Y direction). User U1's head is facing forward (in the +Y direction). User U1's movement direction, head direction, and the shooting direction (DC1) of the front camera (e.g., C1) are consistent. In State A, if the recognition is prioritized based on user U1's movement direction, control can be implemented in a way that prioritizes the capture and recognition of the image from the front camera (C1). Furthermore, for example, if it is desired to recognize objects to the left (-X) of user U1, images from the front and rear cameras with a sufficiently large field of view can be used (in combination). Alternatively, images from the left and right cameras of the smart glasses 1C can also be used.

[0181] As another example, in state B, user U1 moves to the left (-X direction). User U1's head is facing forward (+Y). The direction of user U1's movement is not the same as the shooting direction of the front camera (C1). The direction of user U1's head is the same as the shooting direction of the front camera (C1). In state B, if the setting prioritizes user U1's movement direction (e.g., leftward) for identification, images from both the front and rear cameras with a sufficiently large field of view can be used (in combination). Alternatively, images from the left and right cameras of the smart glasses 1C can also be used. Furthermore, in state B, if it is desired to prioritize head direction for identification, the image from the front camera (C1) can be used.

[0182] As another example, state C is a case where there are cameras in front, behind, left, and right of user U1's smart glasses 1C. User U1 moves forward (+Y), but their head is facing diagonally forward to the left. The direction of user U1's head (diagonally forward to the left) is consistent with the shooting direction of the front camera (C10). The direction of user U1's movement (+Y) is inconsistent with the shooting direction of the front camera (C10) (diagonally forward to the left). In addition, the shooting direction of the right camera (C30) becomes diagonally forward to the right. The direction of user U1's movement (+Y) is also inconsistent with the shooting direction of the right camera (C30) (diagonally forward to the right). Although there is no camera that directly shoots at the forward direction (+Y) as the direction of movement, there are cameras with adjacent shooting directions that are as close as possible to that forward direction, such as the front camera (C10) and the right camera (C30). In state C, with the setting of prioritizing the recognition of user U1's movement direction (e.g., forward), it is sufficient to use the images from the front camera (C10) and the right camera (C30) with a sufficiently large field of view. For example, an image obtained by combining the right side of the image from the front camera (C10) and the left side of the image from the right camera (C30) can also be used. Additionally, in state C, if it is desired to prioritize head orientation for identification, the image from the front camera (C10) can be used.

[0183] As another example, state D is in the torso of user U1, for example, clothing 1E ( Figure 6B In the case of front and rear cameras C5 and C6, user U1 moves, for example, forward (+Y). Let's say user U1's head is facing diagonally forward to the left, but their torso and clothing 1E are facing forward (+Y). User U1's movement direction (+Y) is the same as the shooting direction of the front camera (C5). The direction of user U1's head (diagonally forward to the left) is different from the shooting direction of the front camera (C5). The same applies when user U1's smartphone 1C is facing forward. In state D, if the setting prioritizes user U1's movement direction (e.g., forward), the image from the front camera (C5) can be used.

[0184] As in the example of state C above, when it is desired to identify directions different from the camera's shooting direction (e.g., the direction of movement), this can be addressed by using images from a camera with a sufficiently large field of view or by using images from multiple cameras with different shooting directions. Even in the case of a movement direction priority setting described later, this can be addressed by using images from each camera as in the example above.

[0185] Device 1 can also use various sensors to detect the movement direction of user U1, the direction of the head, the direction of the torso, and the direction of each camera in three-dimensional space. For example, it can use the posture detection unit 312 ( Figure 7AThe sensor in device 1 detects the direction of movement of user U1. Device 1 can also utilize the detected directions in control. Furthermore, if device 1 (e.g., smart glasses 1C) has a gaze detection function, the gaze direction detected by the gaze detection function can also be used in control.

[0186] [Control Mode]

[0187] Figure 13 A table summarizing several control modes is shown as an explanatory diagram of the control modes. It has... Figure 1 While the image reading system of device 1 and server device 2 can be equipped with only one specific mode for control, it can also be equipped with multiple modes and switched as needed. For example, Table 1301 shows control modes related to the camera's shooting direction. As examples of control modes, mode 1 (M1) is a mode that performs shooting / recognition only in front of user U1 and device 1. Mode 2 (M2) is a mode that performs shooting / recognition in the front-back direction. Mode 3 (M3) is a mode that performs shooting / recognition in the front-back and left-right directions. For example, regarding... Figure 5A The smart glasses 1C have cameras in four directions: front, back, left, and right. The control mode can be used to set / control which camera to use.

[0188] Additionally, Table 1302 shows the control modes related to the utilization / cooperation of server device 2. As examples of control modes, Mode 1 (MA) is a mode in which no communication with server device 2 is established, and parsing / identification processing is performed locally on device 1. Mode 2 (MB) is a mode in which communication with server device 2 is established, and parsing / identification processing is performed locally on server device 2. Mode 3 (MC) is a mode in which communication with server device 2 is established, and parsing / identification processing is performed locally on server device 2 when communication with server device 2 is possible (or good), and parsing / identification processing is performed locally on device 1 (e.g., simplified parsing / identification described later) when communication with server device 2 is not possible (or deteriorated).

[0189] [Settings Information]

[0190] Figure 14 This table shows an example of the system's configuration information. This configuration can be applied to each user. Furthermore, multiple configurations can be applied to each user separately. User settings related to the image reading function can be configured on the graphical user interface (GUI) screen provided by device 1 or server device 2 of this system.

[0191] This table includes the following settings: #1 "Using a camera", #2 "Reading camera when using multiple cameras", #3 "Camera image capture interval", #4 "Reading interval", #5 "Determination criteria for reading by the main camera", #6 "Determination criteria for reading by the sub-camera", #7 "Reading object priority", #8 "Reading function without change", #9 "Reading function with different performances", #10 "Safe space guidance function", #11 "History retention period", and #12 "Other detailed settings".

[0192] In item #1, "Using a Camera," regarding camera 403 of device 1, the camera used for recording can be set to be on / off. For example, the front, rear, left, and right cameras of smart glasses 1C can be set to be on / off respectively. Alternatively, in the case of using the 360-degree camera described later, it can be set to be on / off. On (ON) is the active state, and off (OFF) is the inactive state.

[0193] In Project #2, "Reading aloud using multiple cameras," when using multiple cameras for audio reading, you can specify which camera's image the audio will be based on. You can also set the priority / order of audio reading for each of the multiple cameras.

[0194] The setting "equal" controls the equal execution of multiple voice readings corresponding to multiple cameras used. Examples of equal control can be given as described above (…). Figure 9A As shown in the figure, the cameras in front, behind, left, and right are set to the same ratio, and the audio is read out in a predetermined order.

[0195] Regarding the setting "Front-side based (main)", the audio is read aloud based on the image from the front camera. Audio reading aloud from other directions is only performed when necessary (e.g., when predetermined conditions are met) via interruption, etc. For example, in... Figure 9C In control B, the front camera becomes the main camera, and the rear camera becomes the sub-camera. Regarding the setting "Rear-based (Main)," audio is read from the image of the rear camera. Audio reading from other directions is only performed when necessary (e.g., when predetermined conditions are met) via interruption, etc. For example, in... Figure 9C In control D, the rear camera becomes the main camera, and the front camera becomes the sub-camera. (To be discussed later.) Figure 23 Specific examples are shown.

[0196] The setting "based on movement direction" is a control executed based on the audio reading of the camera image corresponding to the movement direction of user U1 (as described above). Figure 12(etc.). Regarding the reading aloud from other directions, it will only be done when necessary (e.g., when predetermined conditions are met), through interruptions, etc. Figure 12 In the examples, in state A, the movement direction is forward (+Y), and the audio reading from the image of the front camera in the same direction becomes the basis. In state B, the movement direction is left (-X), and there is a left camera in the same direction as the movement, so the audio reading from the image of the left camera becomes the basis. If there is no camera in the same direction as the movement, images from other cameras that are as close as possible to the movement direction are used for processing. In state C, the movement direction is forward (+Y), and there is no single camera in the same direction as the movement, but images from the front camera or the right camera that are as close as possible to the movement direction are used for processing.

[0197] Regarding item #3, "Camera Image Capture Interval," the first time interval (period, frequency, etc.) for capturing images for audio reading from a time-series image set of the camera's footage can be set. Alternatively, the time interval can be set separately for the camera used in the base (i.e., the main camera) and for cameras used in other situations (i.e., sub-cameras). This capture interval can also be set numerically, for example, it can be set selectively from several stages.

[0198] Regarding item #4, "Reading Interval," a second time interval (period, frequency, etc.) for reading aloud based on captured images can be set. This reading interval can be set numerically or selectively from several stages. For example, it can be set to high (per second), medium (per minute), or low (every few minutes). Furthermore, the time interval can be set separately for the camera used in the base (i.e., the main camera) and for cameras used in other situations (i.e., the sub-cameras). The system's processing load can be adjusted based on the settings in #3 and #34.

[0199] Regarding item #5, "Judgment Criteria for Main Camera Reading," it is possible to set judgment criteria for the main camera related to the reading camera settings in #2. Setting A, "Objects in front of the direction of movement," controls the reading of audio to be as close as possible to objects detected in the user U1's direction of movement / travel. Furthermore, as another control, it is also possible to apply controls to read audio to be as close as possible to all identified / detected objects.

[0200] Setting B, "Approaching Object," prioritizes an object that is approaching the user U1 when it is the object to be read aloud (described later). Additionally, this control allows setting thresholds for the relative distance and speed of the approaching object. Figure 11 ).

[0201] Setting value C, "significant change," is a control that detects features that show significant changes between images and prioritizes the objects corresponding to these significant changes as the subjects for audio reading (described later). Significant change can also refer to, for example, changes / differences between images related to features / feature points within the image that exceed a threshold.

[0202] Setting D. "Specific Object" is the control that detects a specific object and uses it as the object for voice reading. The specific object can be set by the user in other screens during the transition (e.g., from candidates). Examples of specific objects, as described later, include pedestrian crossings, traffic lights, and traffic signs.

[0203] Setting value E. "Danger Detection" is the control that detects dangerous states / objects in the surrounding environment and reads them aloud as the object (described later).

[0204] Regarding item #6, "Interruption Timing of Sub-Camera Reading," the interruption timing (second time interval) of audio reading related to the sub-camera settings corresponding to #2's reading camera settings can be set. In setting value A. "Every Predetermined Time," the interruption period, etc., can be set for the sub-camera. This setting can also be a setting relative to the proportion of the main camera, etc. For example, for... Figure 9C In the example of control B, the ratio of 3 to 1 is set at a 3:1 ratio, which is equivalent to 1 / 4 of the 3 readings on the main camera (front camera) side and 1 reading on the sub-camera (rear camera) side.

[0205] In setting B, "Detection of Approaching Objects," the audio reading is interrupted when an object approaching user U1 is detected. In setting C, "Detection of Significant Changes," the audio reading is interrupted when an object exhibiting significant changes between images is detected. In setting D, "Detection of Specific Objects," the audio reading is interrupted when a specific object is detected. In setting E, "Detection of Danger," the audio reading is interrupted when a dangerous situation / object is detected in the surrounding environment.

[0206] Regarding item #7, "Reading Object Priority," you can set priorities related to the objects and conditions of the audio reading. This priority is particularly relevant to audio readings of various objects that can be identified / detected within an image. For example, you can have the same objects and conditions as the settings A through E in #5 above. You can assign them a priority order or sort them, or you can select a priority level from several levels such as high / medium / low for each object and condition. Additionally, you can prepare default settings for the system in advance; for example, you can set "Danger" as the highest priority and "Approaching Objects" as the second highest priority.

[0207] Regarding #8 "Unchanging reading function", when it is determined that the objects in the images are almost unchanged, it is possible to set whether to read aloud the meaning of the unchanging state (described later).

[0208] Regarding #9 "Different Representations for Reading Aloud Function", when repeatedly reading aloud the same detected object, the text representing that object can be changed to different representations, and whether or not to read aloud can be set (described later).

[0209] Regarding #10 "Safe Space Guidance Function", it is possible to set whether to read aloud a voice to guide the user to a safe space (described later) based on the positional relationship between the user and the object, such as when the object is close to the user, to prevent the user from touching the object.

[0210] Regarding #11 "History Retention Period", the duration for which historical data of the audio reading is retained can be set. The retention period for both device 1 and server device 2 can also be set. Options for the set value include, for example, long-term, 1 year, 3 months, or no retention.

[0211] #12 "Other detailed settings" refers to detailed settings other than those mentioned in #1 to #11. For example, image resolution can be set. Regarding image resolution, for example, it is possible to set it to high (600dpi), medium (350dpi), or low (72dpi).

[0212] [Recorder Functions]

[0213] This system also includes a recorder function related to image reading aloud as one of its features. It can record according to user settings (…). Figure 14 #11), will utilize image-to-speech functionality for processing (including...) Figure 8The historical data of image capture (802), recognition (803), and voice reading (804) is recorded / saved to storage resources (device 1 or server device 2). This recorder function (in other words, a walking recorder) functions like a dashcam in a car. That is, this function records the user's surroundings as history while walking, and this history can be reviewed later. The historical data saved through this function can be output later (screen display, audio output). This function allows the surrounding conditions at the time to be used as evidence for verification. The historical data may also include date and time, captured images, and text of recognition results.

[0214] [Object identification and audio reading within an image]

[0215] Figure 15A This is a schematic diagram illustrating an example of object recognition and voice reading within an image from camera 403 of device 1. Figure 15A The image illustrates an example of object recognition from a captured image using AI 20, and an example of changes in objects within the image. Specifically, this example shows an object approaching user U1 (i.e., device 1). The upper part is image 1501 showing the state at time point 1, and the lower part is image 1502 showing the state at time point 2. Object OB1 within the image is a pedestrian walking towards the user. AI 20 identifies such an object / situation / change as a feature from the image and transforms it into text representing that object / situation / change. AI 20 uses text to represent the inference result of what object OB1 is.

[0216] First, pedestrians were identified / detected from image 1501 ( Figure 10 In the case where pedestrian 1001 is the object OB1, it can also be represented as text, for example, transformed into "pedestrian" (or "person", etc.), and the "pedestrian" can be read aloud. Within image 1501, region r1 is a rectangular image region that includes object OB1 (OB1a). Point p1 is an example representing the position coordinates of object OB1 / region r1. The AI20 of server device 2 outputs information such as the position coordinates, size, shape, and text of the object OB1 representing such a region as the object OB1 as a response.

[0217] Furthermore, if it is determined that an object OB1 is approaching the user between images, text indicating the approach of the object OB1 can be obtained, such as "pedestrian approaching," and read aloud.

[0218] Within image 1502, region r2 is a rectangular image region encompassing object OB1 (OB1b). Point p2 is an example representing the position coordinates of object OB1 / region r1. For instance, within image 1502, the position (point p2) of region r2, representing a pedestrian as object OB1, relative to the position (point p1) of region r1 in image 1501, has shifted, and the size change is more significant. Figure 15A In the example, the size variation is large.

[0219] Device 1 or server device 2 can also determine whether object OB1 is approaching the user based on the amount of change in the size of the regions (r1, r2) of object OB1 between images. Alternatively, device 1 or server device 2 can also, for example, Figure 11 As shown, the relative distance and speed of the object to the user are determined based on the image. If the relative distance is within a predetermined range (threshold), it is determined to be an approximation.

[0220] Furthermore, the determination of approximation is not limited to binary determination of whether it is approximation; it can also be determined by multi-valued determination of the degree of approximation.

[0221] In addition, Figure 15A In the example, the audio reading of "pedestrian" or the situation of a pedestrian approaching can be performed at time point 1501 of the image, or at time point 1502 of the image.

[0222] This system prioritizes reading aloud from objects that are closest to the user. In other words, it prioritizes reading aloud from objects that are most close to the user.

[0223] Figure 15B Other examples are shown. In image 1511, the object OB2 (OB2a) to be detected is a pedestrian moving to the right from the left side of the intersection. In image 1512, the object OB2 (OB2b) to be detected is a pedestrian arriving at the center of the intersection. Compared to image 1511, in image 1512, the size of region r3 of object OB2 remains almost unchanged as shown by region r4, but the position coordinates change significantly from p3 to p4. Device 1 or server device 2 can also determine the movement of object OB2 based on the amount of change in the position coordinates of regions (r3, r4) of object OB2 between images.

[0224] like Figure 15A As shown, it can determine the object's approach to the user in the front-to-back direction based on changes in the size of the feature. For example... Figure 15B As shown, it can understand the situation such as an object moving relative to the user in the left or right direction and crossing in front of them, based on the changes in the position coordinates of the features.

[0225] Device 1 or server device 2 can use the changes in the size, position, etc., of features corresponding to the object between the images described above to determine the positional relationship between the user and the object. Based on this determination, predetermined control can be performed. Examples of predetermined control include determining the object to be read aloud when the changes in the object are large, or prioritizing the object to be read aloud when the degree of proximity is high.

[0226] Device 1, when controlled to prioritize reading aloud objects close to the user's location, can also adjust the reading based on the relative distance to the user; in other words, based on... Figure 11 The distance range is used to determine the priority of reading aloud. For example, multiple stages can be set as distance thresholds, such as 1m, 3m, 5m, and 10m. For instance, if there are objects approximately 10m and 5m away from the user, the closer object is given higher priority for reading aloud and is read aloud first.

[0227] As a variation, this system can also control the volume of the audio reading by increasing the distance between the object and the user (or the object at a smaller relative distance). Alternatively, it can be set to the relative speed at which the object approaches the user. Figure 11 If the volume is larger, the sound volume will be even louder.

[0228] Furthermore, this system is not limited to volume control based on the degree of approximation; it can also be used to control the timbre. For example, when the degree of approximation exceeds a threshold, the timbre of the voice reading can be changed from timbre 1 to timbre 2.

[0229] In addition, this system can also be controlled by adding warning sounds or vibrations based on the degree of proximity. The output cycle of the warning sound can also be shortened according to the degree of proximity. For example, when the degree of proximity exceeds a threshold, a warning sound or vibration can be added to the reading, or the reading can be changed to a warning sound or vibration.

[0230] In addition, when a non-visually impaired user uses smart glasses 1C, certain displays accompanying the voice reading can be added to the display surface of the device 1, such as displays indicating that an object is approaching.

[0231] Alternatively, the aforementioned degree of approximation can be interpreted as the degree of danger described later. That is, the greater the degree of approximation, the greater the degree of danger can be considered, and corresponding audio readings and warning outputs can be applied accordingly.

[0232] The activities and changes of objects (corresponding features) are not limited to the examples mentioned above. For example, when a new object appears and is detected in the user's field of vision (within its corresponding image), or when the object's direction of movement changes, it also becomes the subject of the audio reading (described below). In addition, changes in object recognition performed using AI20 also become the subject of the audio reading.

[0233] Significant changes in objects between images

[0234] Device 1 or server device 2 is controlled to identify / detect portions of the surrounding conditions / objects that show significant changes over time based on a group of captured images, and to prioritize these portions for audio reading. Based on the recognition results between images, Device 1 or server device 2 identifies / detects portions of the surrounding conditions that show significant changes between the surrounding conditions at a first time point (which is earlier in time) and a second time point (which is later in time) based on the amount of change in features. For example, portions where different activities are taking place are detected as objects with significant changes. Furthermore, Device 1 or server device 2 is controlled to prioritize objects corresponding to these significantly changed portions as objects for audio reading over other portions (i.e., portions that show almost no change).

[0235] As a simple example, the audio reading of objects with significant variations can also be set to output only the names of objects. For example, it could be words like "person," "pedestrian," "bicycle," or "vehicle." When set to short text such as words, a large number of objects can be read aloud per unit of time. More specifically, the audio reading can also be set to output statements describing the state of objects. For example, it could be statements like "a person is approaching," "a bicycle is approaching from behind," or "a car is crossing." When set to statements, users can easily understand the surrounding situation.

[0236] In addition, when this system reads aloud objects that have undergone significant changes, it can also be configured to output audio based on how the object changes, representing the text before and after the change. For example, it can output audio such as "A stopped car suddenly starts moving towards this direction" or "The signal changes from red to green."

[0237] When the audio is set to read aloud objects that change significantly, but not parts that do not change significantly, the same objects that remain almost unchanged will not be read aloud repeatedly. Therefore, for the user, the amount of audio information is reduced, the complexity is decreased, and it is easier to grasp the surrounding situation.

[0238] Figure 16AExamples of significant changes in objects within and between images from camera 304 are shown. First, in image 1601 at time point 1, no particularly prominent object (person) is detected on the sidewalk in front of the user. Then, in image 1602 at time point 2, a pedestrian appears as object OB3 on the sidewalk in front of the user. This object OB3 appears in the image, for example, from the left side of the road onto the sidewalk where the user is located. Device 1 or server device 2 detects these change points and differences between images, and based on these change points and differences, detects objects that have undergone significant changes. Furthermore, device 1 or server device 2 prioritizes the detected objects with significant changes as the objects for audio reading. Figure 16A In the example, the text and audio reading of the recognition result of object OB3 are, for example, "pedestrian" or "pedestrian approaching from the left".

[0239] Figure 16B Other examples are shown. This example illustrates a significant change in the direction of movement of an object between images. In image 1611 at time 1, the pedestrian's direction of movement, as object OB4 (OB4a), is to the right (+X). In image 1612 at time 2, the pedestrian's direction of movement, as object OB4 (OB4b), changes towards the direction closer to the user (-Y).

[0240] This system also determines the amount of change in the direction of movement of the object corresponding to the features of the identification result. Device 1 or server device 2 detects the points of change and differences between such images, and based on these points of change and differences, detects the object OB4 as having significant changes. Furthermore, device 1 or server device 2 prioritizes the detected object OB4 as the object for audio reading. Figure 16B In the example, the text and audio reading of the recognition result of object OB4 are, for example, "pedestrian" or "a pedestrian approaching from the left turns towards you".

[0241] Furthermore, regarding the control example described above, if the judgment is made by the server device 2 through parsing processing instead of device 1, the response information from the server device 2 to device 1 may include not only the text of the identification result, but also the judgment result (such as distance information to the object).

[0242] [Processing Sequence]

[0243] Figure 17 Show Figure 1Example of the processing sequence of device 1 and server device 2 in the system. In step S101, device 1 turns on the voice reading function. For example, it can be turned on automatically according to the settings when device 1 starts up, or it can be turned on by user input. Alternatively, device 1 can determine the posture and other states of device 1 and turn on the voice reading function automatically.

[0244] In step S102, device 1 reads the setting information of user U1 from non-volatile memory 303, etc. Figure 14 Additionally, in step S102, device 1 can also determine the data to be parsed based on the read user setting information. Figure 4 The rules for steps S3 and S4 (also referred to as parsing rules) or the control modes corresponding to those parsing rules, etc. For example... Figure 14 When explaining the setting of #5 ("Judgment Criteria for Main Camera Reading"), the parsing rules are as follows. For example, the following parsing rules can be cited as judgment criteria: parsing is performed according to these priorities / orders, where "B. Approaching object" is given first priority, "D. Specific object" is given second priority, and "A. Object in front of the direction of movement" is given third priority. Figure 4 (The processing of steps S3 and S4 in the process). Additionally, regarding control modes, examples such as... Figure 13 Select from the control modes.

[0245] In step S103, the device 1 starts using the image reading function, turns on the image input of the camera 403, and inputs the image (time-series image frames) of the camera 403.

[0246] In step S104, device 1 begins communication for cooperation with server device 2. Device 1 uses communication unit 330 ( Figure 7A In step S104, device 1 establishes a communication connection with server device 2. Upon establishing this connection, device 1 sends the user configuration information (or information obtained from processing it) from step S102 to server device 2. Additionally, device 1 may also send user information of user U1, device information of device 1, and other information to server device 2 at this time. Furthermore, if device 1 determines the parsing rules and control mode in step S102, it may also send the parsing rules and control mode information to server device 2 in step S104.

[0247] In step S105, server device 2 uses communication unit 430 ( Figure 7BThe server 2 communicates with device 1 to cooperate, receiving user settings, parsing rules, control modes, and other information from device 1. Based on this information, the server 2 determines the parsing processing to be performed in the server device 2. Figure 4 The content of steps S3 and S4 in the process. In step S105, if there are no parsing rules or control modes in the information received from device 1, the parsing rules and control modes may be determined by server device 2 based on user setting information from device 1, rather than by device 1.

[0248] In step S106, device 1 monitors for any operation input from user U1 and determines whether there is a stop instruction, etc. A stop instruction is an indication of termination or temporary cessation related to the image reading function (particularly a series of processes from image capture to audio reading). If a stop instruction is present, device 1 sends the stop instruction to server device 2 and stops the processing in device 1. In step S107, server device 2 stops the corresponding processing in server device 2 based on the stop instruction from device 1. If no stop instruction is present, the processing from step S108 onwards is repeated in a loop. Furthermore, if processing is temporarily stopped according to a stop instruction, and then a start instruction is input based on user U1's operation input, device 1 and server device 2 resume processing.

[0249] In step S108, device 1 waits for a predetermined time corresponding to the user's settings. In this embodiment, device 1... Figure 14 According to setting #3 (first time interval), images are captured at predetermined intervals, and according to setting #4 (second time interval), audio is read aloud at predetermined intervals. Step S108 is the waiting period for image capture. Capture can be performed in each cycle of the main camera and the sub-camera.

[0250] In step S109, device 1 captures / obtains an image from the input image of camera 403 and stores it in memory. When using multiple cameras, the timing of image capture by each camera can also be controlled according to a predetermined schedule, such as... Figure 9A Capture sequentially as shown. Figure 7A The image processing unit 305 controls this image capture and stores the acquired images in the image storage unit 306. Alternatively, predetermined hardware can be provided to allow simultaneous capture using multiple cameras if there are no processing load issues. Furthermore, during image capture, image information such as the shooting date and time, image recognition information, and camera recognition information are added to each image data.

[0251] In step S110, device 1 confirms / determines the communication status with server device 2. Device 1 determines whether communication with server device 2 is possible or whether the communication is good. If the communication status with server device 2 is not possible / bad, device 1 temporarily disconnects communication with server device 2, or temporarily stops sending requests while maintaining the communication connection. Examples of a communication status not possible / bad include situations such as congestion of communication network 9 or high processing load on AI20 of server device 2. When device 1 disconnects / stops communication with server device 2, it saves the previous processing history to memory.

[0252] In step S111, server device 2, based on the communication status between device 1 and server device 2 in step S110, may temporarily disconnect communication with device 1 or temporarily stop sending responses while maintaining the communication connection, as needed. When communication with device 1 is disconnected / stopped, server device 2 saves the previous processing history to memory (DB). This history is retained for a predetermined period. Based on the communication capability / inability status in steps S110 and S111, the system creates a control flag and performs processing from step S112 onwards according to the flag.

[0253] In this system control, if communication is unavailable / malfunctioning in step S110, the parsing process, including the identification process of AI20, is not performed on the server device 2 side. Instead, parsing processing is performed on the device 1 side in step S112. If the device 1 has a function to perform parsing processing on the device 1 side installed, device-side parsing processing can be performed in step S112. If the device 1 does not have a function to perform parsing processing on the device 1 side installed, device-side parsing processing cannot be performed in step S112, so both device 1 and server device 2 stop parsing processing. In this case, for example, device 1 may notify user U1 of the current status and wait for communication with server device 2 to resume.

[0254] In step S112, if a simplified parsing function is installed on device 1, simplified parsing can also be performed on device 1. Normally, parsing is performed on server device 2, which has abundant computing resources. However, if parsing on server device 2 cannot be utilized due to communication limitations, simplified parsing is performed on device 1 (described later). Generally, computing resources on device 1 are more limited than those on server device 2 in the cloud computing system 3. Therefore, simplified parsing on device 1 refers to a process that is simpler than parsing on server device 2. When computing resources are abundant on device 1, such as... Figure 3 As shown, it is also possible to perform parsing processing using only device 1.

[0255] In this system control, if communication is enabled / good in step S110, in step S113, the parsing processing, including AI20 identification processing, is performed not on the device 1 side, but on the server device 2 side. In step S113, for the parsing processing on the server device 2 side, device 1 sends a parsing request and an image (the captured image and additional image information from step S109) to server device 2. Furthermore, when sending this request, supplementary information may be added in addition to the image. Supplementary information is information that can be used for parsing processing; examples include information that can be obtained from the sensors of device 1 (e.g., [insert example here]). Figure 7A The information detected by the position detection unit 311 and the posture detection unit 312 includes, for example, position, posture, orientation, acceleration, etc.

[0256] In step S114, server device 2 receives a parsing request and an image from device 1 and stores them in memory. Based on the request, server device 2 begins parsing processing, including recognition processing using AI 20. First, server device 2 uses AI 20 to recognize the image and extracts features related to the object from the image. In other words, it detects the object from the image. Furthermore, server device 2 uses this recognition to transform the object corresponding to the feature into text representing the object. As output of AI 20, feature information including text information corresponding to the feature, position coordinates, size, etc., is obtained. The processing of each captured image described above is repeated at predetermined intervals.

[0257] Next, in step S115, the server device 2 detects the changes (in other words, points of change) between the images at the current time point and the images at the previous time point in the captured image group over the time series. These points of change include, for example, changes in the position coordinates, size, shape, orientation, etc., of the object (in other words, the image region) corresponding to the features.

[0258] Next, in step S116, server device 2 determines the distance, speed, and direction of change (in other words, direction of movement) of objects within and between images and user U1 based on the application's parsing rules, control mode, etc. Server device 2 can also determine distance, speed, and direction of change through image parsing. Furthermore, server device 2 can also determine significant changes, proximity, danger level, and safe space related to objects. In this embodiment, this determination is performed on server device 2 in step S116, but it can also be performed on device 1 after obtaining object information from server device 2.

[0259] In step S117, server device 2 determines the image, object / surrounding situation, text, etc., as the object to be read aloud, based on the results of the parsing and judgment processes up to step S116. Server device 2 compares features from the image, as well as points of change, distance, speed, and direction of change, with the conditions in the parsing rules, such as according to parsing rules and control modes. Figure 14 The priority of #7 determines the object to be read aloud, and also determines the text corresponding to the determined object.

[0260] In step S118, server device 2 sends a response, including information such as the text of the object to be read aloud as determined in step S117, to device 1 as a parsing result. Alternatively, when sending a response from server device 2 to device 1, processing result information from the parsing and determination processes on server device 2 (e.g., priority / order of voice readings related to multiple objects) can be added according to parsing rules / control modes, etc. In step S119, device 1 receives the response from server device 2, converts the text of the object to be read aloud and the additional information into sound, and outputs the sound from a sound output device such as a speaker, thereby performing voice reading. When the text about the object has a priority / order, device 1 controls the voice reading accordingly. For example, device 1 reads multiple texts in order of priority. Additionally, for example, device 1 changes the volume of the voice reading based on the degree of proximity. Furthermore, for example, device 1 outputs a warning sound from a sound output device when a warning output indication is provided.

[0261] The audio reading in step S119 is set to be performed at predetermined intervals (the second time interval). Therefore, the system is controlled to perform the processing of steps 112 to S119 according to the intervals. Furthermore, when there is no object to be read during the periodic timing of the audio reading, for example, when there is no significant change, the response of step S118 is that there is no object to be read (meaning no change), which is a response of no object text.

[0262] in addition, Figure 17The sequence is an automatic, periodically executed series of processes from image capture to audio reading. However, if a reading instruction (re-reading instruction described later) is input by user U1 at any given time, the control can be performed as follows: When device 1 receives a reading instruction, and communication with server device 2 is enabled / good, it captures an image and sends the request corresponding to the reading instruction, along with the image, to server device 2. Server device 2, in turn, performs parsing processing based on the request and sends a response including the parsing results to device 1. Device 1 then reads the text aloud based on the response.

[0263] In this embodiment, as in the example above, multiple cameras on device 1 automatically perform a series of processes from image capture to audio reading at predetermined time intervals. This largely eliminates the need for user-instructed operations such as image capture, resulting in minimal workload and high convenience.

[0264] Provided time and computing resources permit, this system can also read aloud all objects identified from an image. However, even reading a large number of objects at once can be difficult for users to understand and may cause confusion. Therefore, in this embodiment, during the automatic image reading cycle, the objects to be read are determined according to priority, proximity, etc., and the important parts are read aloud first. This provides appropriate assistance that is easy for users to understand. Users can distinguish between user-defined information and control modes based on the usage scenario, the app, etc.

[0265] Furthermore, in a time-series context, when no object is detected or the object remains almost unchanged, there is a possibility that the number of audio readings may decrease, or the same audio readings may be repeated. In such cases, the user may feel uneasy. Therefore, this embodiment also includes the function of providing audio readings at arbitrary time intervals via input reading instructions, and, as described later, providing readings with the same meaning even when the surrounding conditions remain unchanged. This provides appropriate assistance that is easy for the user to understand.

[0266] As Figure 17 In a variation, in step S118, server device 2 may also convert the text representing the object into sound data for reading aloud and send a response including the sound data to device 1. In this case, the text-to-sound conversion processing in device 1 is not required.

[0267] [Basic Controls]

[0268] Figure 18This diagram illustrates the basic control related to automatic image capture and voice reading over a time series. For example, time points 1 to 7 are shown as the voice reading cycle timing 1801. The image 1802 at each time point is an example of an image captured under timing corresponding to the time point of the voice reading cycle. Voice reading 1803 represents an example of the audio content read at that time point. Repeat 1804 indicates the number of times the voice reading is repeated over the time series.

[0269] Let's say at time point 1, no object was specifically detected in the image. At time point 1, no audio was read aloud (illustrated as "None"). At time point 2, for example, with... Figure 16A Similarly, as an example of significant change, pedestrian 1811, as object 1811, was detected. At time point 3, the state was that pedestrian 1811 was in front of user U1 on the sidewalk. At time points 4 and 5, the state was the same as at time point 3, with pedestrian 1811 showing almost no change. At time points 6 and 7, the state was that pedestrian 1811 was moving in a manner approaching user U1. At each time point after time point 2, "pedestrians" as object 1811 were detected within the image. Therefore, at... Figure 18 In the control example, at each time point after time point 2, the same sound, such as "pedestrian" (or specifically "pedestrian approaching"), is output repeatedly. Time point 2 is the first repetition, and time point 7 is the sixth repetition.

[0270] As in the example above, even without any special input, user U1 can identify "pedestrians" ahead based on the voice reading at various points in time.

[0271] Regarding the control described above, device 1 saves historical data, including image capture and audio reading processing, to storage resources. Device 1, for example, retains this historical data in the memory of its own device or server device 2 until a predetermined time limit, number of times, or data volume is reached. If the historical data exceeds the storage limit, device 1 or server device 2 sequentially rewrites and deletes older historical data.

[0272] [Do not control the voice reading if there are no changes]

[0273] then, Figure 19 Is based on Figure 18 This is an explanatory diagram related to the basic control of not reading aloud when the object remains unchanged. An example of a "pedestrian" as the object 1811 is shown. Figure 18The same applies. During the transition from time point 3 to time point 4, in other words, between these images, the "pedestrian" as object 1811 remains almost unchanged. Therefore, in this control example, at time point 4, device 1 does not perform audio reading (illustrated as "none"). Similarly, during the transition from time point 4 to time point 5, there is no change, so no audio reading is performed.

[0274] The image reading system can also determine that the object has not changed if the amount of change between images (such as the change in the position coordinates of features, size, etc.) is less than a threshold, based on the recognition results of AI20.

[0275] During the transition from time point 5 to time point 6, the "pedestrian" (object 1811) approaches user U1. Based on the amount of change between images, the system determines that the "pedestrian" (object 1811) has changed. Therefore, at time point 6, regarding this changed object 1811, the system outputs a sound such as "pedestrian" (or specifically, "pedestrian approaching"). Considering the number of repetitions from time point 2, time point 6 marks the third repetition. Similarly, at time point 7, a change is determined, so a fourth sound is output.

[0276] As in the example above, if the object remains unchanged and there is no audio reading, then the amount and frequency of sound heard by user U1 will decrease.

[0277] [Controlling the pronunciation of a word without any change in meaning]

[0278] exist Figure 18 , Figure 19 For example, between time point 3 and time point 5, the objects surrounding user U1 remain almost unchanged. In such a situation where the objects remain unchanged, it is also possible to... Figure 20 Similar to a control example, the system periodically reads aloud a voice indicating no change to the user. For example, the system determines whether a state where the degree of change (amount of change, etc.) of an object is less than a threshold has persisted for a predetermined time. The time count can be performed in units such as predetermined periodic timing or images. The predetermined time for this determination can be set. If the state persists for a predetermined time, the system decides to read aloud a voice indicating no change. In device 1, the voice indicating no change is read aloud, for example, "No change."

[0279] exist Figure 20In the example, at time points 2 and 3, a change is detected in the "pedestrian" (object 1811) in front of user U1, so the voice reading "pedestrian" is performed as described above. At time point 4, it is determined that the "pedestrian" (object 1811) has not changed. If this state of no change lasts for one cycle of the voice reading, it is determined that the voice reading means "no change." Therefore, at time point 4, the voice reading "no change" is performed. Similarly, even at the next time point 5, it is determined that the "pedestrian" has not changed, and this state of no change lasts for one cycle, so the voice reading "no change" is performed.

[0280] In other examples, where the predetermined time for which the unchanging state lasts becomes two cycles of the reading aloud, at time point 4, there is no reading aloud, and at time point 5, the unchanging state lasts for two cycles, so the reading aloud is "no change".

[0281] As in the example above, if the object remains unchanged, the meaning is read aloud without any change, so user U1's anxiety about the lack of audio reading is reduced.

[0282] [Function to read the instructions aloud again]

[0283] Figure 21 This is an explanatory diagram showing a specific example related to the function of a read-aloud instruction. User U1 can input a read-aloud instruction (in other words, a mandatory read-aloud instruction) to device 1 at any given time. The input for the read-aloud instruction made by user U1 can be, for example, based on using a microphone, etc. Figure 7A The method of audio input in the microphone / audio input unit 316 can also be... Figure 5A The remote control 1c and other devices input predetermined buttons. For example, pressing a predetermined button on the remote control 1c indicates that the instruction is read aloud again.

[0284] When the instruction to read aloud again is input, device 1 will always perform a sound reading of the object at that time, outputting a certain sound. Based on the input of the read-aloud instruction, device 1 captures an image from the input image, enabling AI 20 to recognize the image and perform a sound reading about the object / surrounding situation. Even if the object remains unchanged at the time the read-aloud instruction is input, device 1 will still perform a sound reading informing the user of the object or its surrounding situation.

[0285] Figure 21 Images at various time points and Figure 18The changes are roughly the same at time points 2 and 3. As a significant change, a "pedestrian" as object 1811 is detected, and device 1 automatically reads out the word "pedestrian" aloud. After time point 4, the "pedestrian" as object 1811 remains almost unchanged. At time point 4, it is determined that the "pedestrian" as object 1811 has not changed, so if there is no action from user U1, device 1 does not read out the word. At time point 5, similarly, it is determined that there is no change, so if there is no action from user U1, device 1 does not read out the word.

[0286] On the other hand, for example, suppose that immediately before time point 5, user U1 inputs a re-reading instruction. In this case, device 1, according to the re-reading instruction, performs a voice reading about object 1811 at time point 5. In device 1, the voice reading is, for example, "pedestrian". Or, as a variation, with Figure 20 Similarly, since the object remains unchanged, device 1 can also output "no change" or similar.

[0287] At time point 6, it is similarly determined that object 1811 has not changed, so if there is no action from user U1, device 1 will not perform audio reading. At time point 7, user U1 inputs a read-aloud instruction again. In this case, device 1, in accordance with the read-aloud instruction, performs audio reading at time point 7, for example, outputting "pedestrian" (or "no change").

[0288] In the above example, the timing of the input re-reading instruction, as shown at time points 5 and 7, is roughly the same as the periodic timing of the voice reading, but these timings can also be different. In the above example, basically, at time points 5 and 7, device 1 sends a request to server device 2 based on the re-reading instruction, receives a response from server device 2, and uses the response to perform voice reading. Device 1 re-performs voice reading based on the current identification even if the object has not changed relative to the previous identification and the result of the voice reading.

[0289] Not limited to this, as a variation, device 1 may also omit exchanging information with server device 2 when it is given a read-aloud instruction (especially when it is not timed periodically with the voice reading), and use the result of a previous voice reading (e.g., the same text) that has been kept as history to perform voice reading.

[0290] As in the example above, unlike the automatic voice reading using this system, user U1 can immediately perform voice reading at a desired time to confirm the surrounding situation. Furthermore, the aforementioned function of indicating re-reading is similar to the following... Figure 22The combination of different voice reading functions is also effective.

[0291] [The Functions of Voice Reading in Different Manners (1)]

[0292] As shown in the control examples above, the system automatically performs repeated audio readings over a time sequence, either periodically or in response to user U1's instruction to read again. This audio reading scheme, as described in the control examples, can be implemented when the surrounding conditions / objects are largely the same at each time point without significant changes. While the same audio output can be repeated, other embodiments are possible, as described below. In other embodiments, when repeatedly performing audio readings about the same surrounding conditions / objects, the system can be controlled by using different text for audio readings, allowing for explanations from different perspectives and expressions each time.

[0293] Figure 22 This is an explanatory diagram showing specific examples related to the function of voice reading under different performances. Let's assume... Figure 22 The state of objects within the image and Figure 16B Same. Figure 22 In the image 2200, an object 2201 is detected approaching user U1 from the left. The condition of object 2201 remains almost unchanged across the time-series images. Device 1 is configured to enable voice readings for different appearances. For example, at time point 1, based on the detection of significant changes in the response from server device 2, device 1 outputs the first automatic voice reading, for example, "Pedestrian approaching from the left."

[0294] Next, set time point 2 as the timer for the predetermined period of audio reading, or user U1 and... Figure 21 Similarly, the timing of the operation input for the rereading instruction was performed. At this time point 2, device 1 performed a second voice reading about the "pedestrian" as object 2201. Device 1 is controlled to perform a voice reading with a different expression than the previous (first) reading when repeatedly reading about the same object 2201 that remains unchanged. Based on the response from server device 2, device 1 outputs a sound such as "the person is diagonally in front to the left" during the second voice reading.

[0295] Next, time point 3 is set as the timing of a predetermined period of voice reading, or the timing of an operation input by user U1 indicating to read again. At time point 3, device 1 performs the third voice reading about object 2201. Device 1 is controlled to perform a voice reading with a different expression than the first and second readings. Based on the response from server device 2, device 1 outputs a sound such as "a person is approaching from the left road" during the third voice reading.

[0296] To achieve this functionality, device 1 can also use AI20 to identify text again during each scheduled audio reading, obtaining text with different representations from AI20. Alternatively, device 1 can obtain and store multiple representations of text from AI20 during a single request-response process, and then select from these multiple representations for audio output during each audio reading.

[0297] As another specific example, as described below. Suppose that in image recognition, a situation is detected where a man is pushing a bicycle in front of a user and approaching the user, as an object. At time 1, the first audio reading is, for example, "man and bicycle". At time 2, in the second audio reading, as a different expression, it is, for example, "man pushing bicycle". At time 3, in the third audio reading, as a further different expression, it is, for example, "the bicycle pushed by the man is approaching".

[0298] As in the example above, there can be multiple expressions for describing the same surrounding situation / object. This function reads the sentences aloud with different expressions at different points in time. This makes it easier for the user to understand the situation.

[0299] [The Functions of Voice Reading in Different Manners (2)]

[0300] Furthermore, regarding the voice reading function under the different manifestations mentioned above, as a variation, the following control can also be performed: This system can also, at each timing of voice reading, such as automatic cycles or rereading instructions, add or synthesize the content of the current recognition result to the voice reading content of the previous recognition result, and then perform voice reading. In other words, this system can also, when it is determined at each timing that the object has changed, perform voice reading based on both the sound representing the object from the previous time and the sound representing the changed part from the current time.

[0301] As an example, at time point 1, based on the first rereading instruction, AI 20 identifies a "pedestrian" as an object in front of user U1 and reads the sound of "pedestrian". Then, at time point 2, based on the second rereading instruction, AI 20 identifies a "bicycle" (or "person riding a bicycle") as a different object near the previous "pedestrian". In this case, device 1 outputs a sound combining the previous "pedestrian" and the current "bicycle", for example, "pedestrian and bicycle". Next, at time point 3, based on the third rereading instruction, AI 20 identifies a "dog" as another object near the previous "pedestrian" and "bicycle". Device 1 outputs a sound combining these objects, for example, "person, bicycle, and dog". In this example, multiple nouns of objects are linked together and read aloud, but it is not limited to this. More specifically, it can also be set as a statement that describes the state of more than one object, such as "a person riding a bicycle is leading a dog".

[0302] [The Functions of Voice Reading in Different Manners (3)]

[0303] Furthermore, as a variation, the system can also achieve the following: At various time points during automatic repetition or re-reading instructions, the system can read out the voice of different objects based on updates / corrections made using the AI20's recognition. For example, at time point 1, based on the first re-reading instruction, if a "pedestrian" is detected in front of user U1, the system outputs the voice of "pedestrian." At time point 2, based on the second re-reading instruction, the AI20's re-recognition result is updated / corrected regarding the previous object "pedestrian," more accurately detecting it as "a person riding a bicycle," and the system outputs the voice of "a person riding a bicycle." Furthermore, at time point 3, based on the third re-reading instruction, the AI20's re-recognition result is updated / corrected regarding the previous "person riding a bicycle," more accurately detecting it as "a person riding a bicycle with a dog," and the system outputs the voice of "a person riding a bicycle with a dog."

[0304] Furthermore, as described above, when device 1 reads aloud the content based on the correction from the previous reading, it can also read aloud the meaning of the correction to the user during the current reading. For example, in the second reading, it could output a voice message such as "Correction: It was not a pedestrian, but a person riding a bicycle."

[0305] [Example of front and rear camera control]

[0306] Figure 23 Shown in such Figure 9A The example shown illustrates the control of the front and rear cameras when using multiple cameras. Figure 23 Specifically, an example of switching between using the front camera as the main camera and using the rear camera as the main camera is shown. This control enables functions such as interruption. For example, similar to the above... Figure 13 The control mode (M1) uses both the front and rear cameras accordingly. Figure 23 As an example of a situation, suppose a user U1 is walking on sidewalk 901, with a pedestrian 2301 in front and a bicycle 2302 behind. User U1's device 1 and, for example... Figure 9A Similarly, the smartphone 1A is configured to have a front camera C1 and a rear camera C2.

[0307] In normal operation within this system, Control 1, as illustrated, for example, frequently prioritizes and performs image capture / recognition / reading using the front camera C1 in front of (+Y) view. Control 1 and... Figure 9C Control B corresponds to a processing ratio of 3:1 between the front and rear cameras. When control 1 is active, changes in pedestrian 2301 can be detected with high precision from the image of the front camera C1, and for example, a voice reading such as "pedestrian approaching" can be performed.

[0308] This system automatically switches from control 1 to control 2 based on predetermined judgment results, etc. For example, in control 1, at a timed period such as time point 4, objects in the rear direction relative to user U1 are detected from the image of the rear camera C2. For example, as a significant change, a bicycle 2302 moving towards user U1 can be detected. Device 1 switches to control 2, for example, if an object with a significant change in the rear direction is detected from the image of the rear camera C2.

[0309] The control shown in the diagram, Control 2, is a high-frequency, priority control that utilizes the rear (-Y) of the rear camera C2 for capturing / identifying / reading. Control 2 and... Figure 9C Control D corresponds to a processing ratio of 1:3 for the front and rear cameras. When in control 2, changes in the bicycle 2302 can be detected with high precision from the image of the rear camera C2, for example, by reading aloud a voice such as "bicycle approaching".

[0310] By switching from control 1 to control 2 as described above, the function / effect of interrupting the forward recognition for user U1 and inserting the backward recognition can be achieved. Similarly, automatic switching from control 2 to control 1 is also achieved. When the system detects an object with significant changes in the image from the front camera while in control 2, it switches to control 1. In other words, according to this function, it is possible to interrupt the recognition / sound reading for the main camera and insert the recognition / sound reading for the sub-camera. The same control can be performed even when using front, rear, left, and right cameras.

[0311] Furthermore, as a variation, this system can also, when objects are detected in both directions (front and back), judge and compare the degree of change, priority, etc., regarding these objects to determine which one is more important, or in other words, which object should receive more attention. Based on the judgment result, it selects between Control 1 and Control 2. In the illustrated example, when pedestrian 2301 and bicycle 2302 are detected, based on the relative distance and speed, for example, bicycle 2302 is determined to have a higher priority. In this case, the system switches to Control 2 or maintains Control 2, prioritizing the audio reading about bicycle 2302.

[0312] Alternatively, as a variation, the control could be positioned between control 1 and control 2, as described above. Figure 9A State A Figure 9C As shown in control C, there is control that handles the same actions in the forward and backward directions. For example, this system is normally set to control C, and if an object with a significant change is detected in the image from the front camera, it switches to control B, and if an object with a significant change is detected in the image from the rear camera, it switches to control D.

[0313] Additionally, in variations, it is also possible to... Figure 23 Control 1 is replaced by the above Figure 9C Control A is replaced by Control E, and the system switches between Control A and Control E. However, in this case, for example, if no object with a significantly changed state is detected in the image from the rear camera while Control A is in operation, the predetermined judgment for switching needs to be set to other conditions. An example of such other conditions is that if, while Control A is in operation, the state of an object with a significantly changed state is not detected in the image from the front camera for a predetermined period of time, the system switches to Control E. Alternatively, the system can simply switch between control A and control E based on button input, voice input, or other input made by the user U1.

[0314] [Control to differentiate the thresholds before and after]

[0315] Figure 24shows a control example when a front-rear camera that captures images in the front-rear direction for the user U1 is used to determine a significant change in an object (a case where the object performs a different action compared to a past image. In particular, in Figure 24 this control example, a case is shown in which the thresholds for determination are different for the images and recognition of the front and rear cameras of the device 1. This control example is based on the control related to the relative distance and speed between the user U1 and the object as described above ( Figure 11 ). Figure 24 , for the image from the front camera C1 that captures the front of the user U1, a distance threshold TD1 and a speed threshold TV1 are used as the determination thresholds. On the other hand, for the image from the rear camera C2 that captures the rear of the user U1, a distance threshold TD2 and a speed threshold TV2 are used as the determination thresholds. Figure 24 Situation A is an explanatory diagram related to the determination of the relative distance, and Situation B is an explanatory diagram related to the determination of the relative speed.

[0316] For example, similar to Figure 23 control 1, the front camera C1 is set as the main camera, and the rear camera C2 is set as the sub camera, so that shooting / detection / audio announcement is performed preferentially and at a higher frequency for the front than for the rear. In this case, from the perspective of frequency, objects in the rear are more difficult to detect than objects in the front. Therefore, the thresholds (TD2, TV2) related to audio announcement for objects determined from the image of the rear camera C2 may be set to different values in a manner that makes detection easier than the thresholds (TD1, TV1) related to audio announcement for objects determined from the image of the front camera C1.

[0317] In the example of Situation A, rear priority is set, and the rear distance threshold TD2 is set to a larger value than the front distance threshold TD1 (TD1<TD2). In this example, TD2 is approximately twice TD1. Under Situation A, a pedestrian 2401 travels in the -Y direction from the front of the user U1, and another pedestrian 2402 travels in the +Y direction from the rear of the user U1. On the front side, according to the analysis of the image from the front camera C1, when the distance D2401 to the pedestrian 2401 becomes within the distance threshold TD1, the pedestrian 2401 is determined to be the object for audio announcement. On the rear side, according to the analysis of the image from the rear camera C2, when the distance D2402 to the pedestrian 2402 becomes within the distance threshold TD2, the pedestrian 2402 is determined to be the object for audio announcement. For example, even if the distance D2401 and the distance D2402 are approximately the same, since the distance thresholds are different, the pedestrian 2402 on the rear side is detected preferentially first, and audio announcement is performed (for example, "A pedestrian is approaching from behind").

[0318] In situation B, user U1 is stationary with speed v1=0. Pedestrian 2401 in front is moving towards user U1 in the -Y direction at speed v2A, and pedestrian 2402 behind is moving towards user U1 in the +Y direction at speed v2B. In the example of situation B, rear-side priority is set, and the speed threshold TV1 for the rear is set to a smaller value (TV1>TV2). On the front side, based on the image analysis of the front camera C1, if the relative speed with pedestrian 2401's speed v1A is greater than or equal to the speed threshold TV1, pedestrian 2401 is identified as the subject of the audio reading. On the rear side, based on the image analysis of the rear camera C2, if the relative speed with pedestrian 2402's speed v2B is greater than or equal to the speed threshold TV2, pedestrian 2402 is identified as the subject of the audio reading. For example, even if speeds v1A and v1B are the same, because the speed thresholds are different, pedestrian 2402 on the rear side is detected first and the audio reading is performed. Even when user U1 is moving, the same control can be performed using relative speed.

[0319] In addition, even in such Figure 9A As shown in state A, where recognition / voice reading is performed using front and rear cameras at equal ratios, the same method can be applied. Figure 24 Threshold control. For example, if user U1 is not visually impaired, it is also possible to consider prioritizing objects in front of the user and prioritizing objects behind the user, so as to increase the sensitivity of the threshold for objects behind the user.

[0320] Furthermore, regarding the determination of relative distance and relative speed as described above, either one or both can be used. When used together, for example, the object to be read aloud can be determined by satisfying an AND (logical AND) condition for using two thresholds. Even when using cameras in the front, back, left, and right directions, the same threshold control can be applied in all directions. Various threshold settings for control, as described above, can be made based on the user, the app, the usage scenario, etc. Various thresholds can be variably set through the user setting function, and the mode corresponding to each setting can be selected.

[0321] [Assistance for the visually impaired]

[0322] Figure 25A This is an illustrative diagram illustrating one of the scenarios in which visually impaired individuals can be assisted. For example, it is a YZ plan view showing a visually impaired user U1 standing still or walking on sidewalk 2500, observed from the side. For example, similar to the above... Figure 6BSimilarly, in device 1 such as Hat 1D, there are cameras that capture images in both forward and backward directions (in this example, a front camera C3 and a rear camera C4). The front camera C3 and the like preferably have a shooting range capable of capturing an area including the soles of the user U1's feet. The user U1 walks, for example, using a white cane to tap on the pavement 2500 or similar surface to confirm their location. In situation 1, there is some kind of obstacle 2501 on the pavement in front of the user U1. Even in this situation, for example, based on the image from the front camera C3, the obstacle 2501 is captured, identified as an object by AI 20, and transformed into text (e.g., "obstacle" / "situation where there is an obstacle ahead"). Furthermore, device 1 reads aloud about the object (e.g., "There is an obstacle ahead"). Thus, walking can be supported for visually impaired individuals.

[0323] Furthermore, in situation 2, a person 2502 approaches from behind the user U1. Even in this situation, for example, based on the image from the rear camera C4, the person 2402 is captured, identified as an object by AI 20, and converted into text (e.g., "person" / "situation where a person is approaching from behind"). Then, device 1 reads aloud about the object (e.g., "a person is approaching from behind"). In particular, this system can also control the approach to increase the degree of approach and danger based on the distance and speed of the approaching object, and add additional warnings. For example, outputting a sound like "Please be careful" or a warning tone. This allows for more effective support for the walking of visually impaired individuals.

[0324] For example, in the case of a visually impaired user who cannot clearly distinguish between what is in front and behind them, it is desirable to control multiple cameras to provide both audio reading from the front and audio reading from the back. In this case, for example, it could be done as described above. Figure 9A As shown, it can be applied to controls where the ratios are equal before and after, and can also be used as follows: Figure 23 The control applied before and after the switch is shown.

[0325] [Nighttime Support]

[0326] Figure 25BThis diagram illustrates a scenario related to assisting with nighttime navigation. For example, in nighttime navigation 2510, device 1, acting as user U1, uses images from the front and rear cameras of a smartphone 1A, etc. This system also uses the rear camera C2 to detect whether a suspicious person (human) 2511 is approaching user U1 from behind, and reads aloud to alert them (e.g., "A person is approaching from behind," "Please be careful"). In addition, a warning sound can be output along with the voice reading, device 1 can vibrate, and the volume of the voice reading can be increased. Furthermore, the brightness of the screen of the smartphone 1A, etc., can be temporarily increased. For example, increasing the volume of the voice reading can also achieve a deterrent effect on the suspicious person 2511.

[0327] This system can also determine the degree of suspicion of a suspicious person (person) 2511 detected behind user U1, and adjust the voice reading scheme (e.g., volume, timbre, warning, etc.) according to the degree of suspicion. For example, methods for determining the degree of suspicion can include situations where a person detected behind maintains a certain distance from user U1 while following behind, appearing to be concealing their position.

[0328] In addition, when used for applications such as night road assistance, the camera 403 of device 1 can also be an infrared camera or similar device that can shoot in dark environments.

[0329] [Function to handle "using mobile phone while walking"]

[0330] Figure 25CThis diagram illustrates a scenario related to the "walking while using a mobile phone" function. For example, assuming a generally safe situation such as a sidewalk, user U1, for instance, is holding a smartphone 1A (device 1) and walking while searching for their destination using a map app. Using images from the front and rear cameras of smartphone 1A, the system reads aloud the surrounding objects to user U1. For example, various objects recorded on the map, particularly facilities and intersections along the path to the destination, can be detected and read aloud. Furthermore, to ensure safety while "walking while using a mobile phone," objects approaching the user (people, bicycles, vehicles, etc.) are also detected and read aloud in the same way as described above. In this case, approaching objects are prioritized for reading aloud compared to facilities. In the illustrated example, "Intersection A" and "Convenience Store A" are detected in the direction user U1 is walking, and these are read aloud. Additionally, if a bicycle 2521 approaches user U1, it is detected, and a message such as "Bicycle approaching from the right. Please be careful" is read aloud. This can improve safety when using a mobile phone while walking, and it can also work with map apps.

[0331] [Vehicle Driving Assistance]

[0332] Figure 25D This diagram illustrates a scenario related to driver assistance for vehicles. For example, user U1 rides an electric bicycle 2531. While riding the electric bicycle 2531, a camera 403 on a device 1 such as smart glasses 1C or a smartphone 1A worn by user U1 captures images of the surroundings, detects objects, and reads them aloud. For example, if objects (people, bicycles, vehicles, etc.) are approaching user U1 from the electric bicycle 2531 in the front, back, left, or right directions, the audio is read aloud prioritizing these objects. This assists driving and contributes to traffic safety.

[0333] [Control examples corresponding to whether or not the server can communicate]

[0334] Figure 26 This illustrates control examples corresponding to whether or not the server device 2 can communicate, and shows examples related to the above. Figure 17The control example related to step 110, etc. State A is a control example where communication with server device 2 is possible, and the parsing processing is performed on the server device 2 side. Device 1 (e.g., smart glasses 1C) uses control A1 for cameras (C10, C20, C30, C40) in four directions: front, back, left, and right. Control A1 sets equal proportions in the four directions and sequentially sends requests to server device 2 to perform parsing processing, including AI20 recognition. For example, at time point 1, request 1 to parse the image of the front camera C10 is sent. At time point 2, request 2 to parse the image of the rear camera C20 is sent. At time point 3, request 3 to parse the image of the right camera C30 is sent. At time point 4, request 4 to parse the image of the left camera C40 is sent. This process is repeated from time point 5 onwards, starting with the front camera C10. Server device 2 performs parsing processing based on each request and sends each response. Device 1 performs voice reading based on each response, provided there is a timed and targeted voice reading. As mentioned above, it is also possible to set the ratio and prioritize control based on the camera in a specific direction.

[0335] State B is a control example in a state where communication with server device 2 is not possible, and it represents a case where parsing processing is performed on device 1. Here, it is assumed that device 1 has fewer computing resources than server device 2, in other words, lower computing power, and therefore spends more time on parsing processing. Therefore, the parsing processing on device 1 is simplified compared to server device 2. Simplified parsing processing here means parsing processing that reduces the processing load on device 1. To reduce the processing load, control B1 sets the processing ratio by using only a portion of the four cameras (in the on state). For example, only the front and rear cameras are used, the left and right cameras are turned off, and the front and rear cameras are set to an equal ratio.

[0336] For example, at time point 1, device 1 analyzes the image from the front camera C10. At time point 2, device 1 either pauses the analysis process or continues analyzing the image from the front camera C10 at time point 1. At time point 3, device 1 analyzes the image from the rear camera C20. At time point 4, device 1 either pauses the analysis process or continues analyzing the image from the rear camera C20 at time point 3. This process is repeated after time point 5. Device 1 performs audio reading at each time point where simplified analysis has been performed, provided that the audio reading is timed and there is an object present. In the case of control B1, for example, with... Figure 9A Compared to control 1, the cycle time related to image capture and voice reading is longer, resulting in a lower processing load.

[0337] In this way, even when communication with server device 2 is not possible, user U1 can still be assisted by simplified analysis on device 1. Similarly, control can be performed to limit analysis processing to cameras in a specific direction. Control B2 is an example that is limited to the front camera C10 only. Device 1 analyzes the image of the front camera C10 at the first time point and the third time point. In addition, by setting it to analyze only at the first time point, the processing load can be further reduced.

[0338] As another example of control, it is not limited to a binary state of communication capability / inability with server device 2; it can also be set to control corresponding to a multi-valued state. For example, in this system, the communication performance between device 1 and server device 2 is assessed by knowing the communication speed, and different controls are applied according to the communication speed, etc. For example, normal control can be applied when the communication speed is above a first threshold, low-speed control can be applied when it is below the first threshold, and even lower-speed control can be applied when the communication speed is below a second threshold.

[0339] [Priority of multiple objects]

[0340] Figure 27 This example illustrates a control scheme where multiple objects are detected in the surrounding environment of user U1, and the voice is read aloud based on the priority of the objects. Assume user U1 is walking forward (+Y) on the left side of road 2700. A car 2701 detected in front of user U1 stops. Additionally, pedestrians 2702 and 2703 are detected walking towards user U1 in the -Y direction. Furthermore, a bicycle 2704 is detected moving towards user U1 in the +Y direction. Assume device 1 detects these four objects at approximately the same time.

[0341] Device 1 prioritizes these four objects and reads them aloud in order of priority. Examples of priority determination are as follows: First, for example, car 2701 is stopped, so its relative distance to user U1, based on speed, becomes a low priority. Pedestrian 2702 is approaching user U1, so based on the aforementioned relative distance, speed, and degree of approach, it becomes a high priority. Furthermore, regarding moving pedestrians 2702, 2703, and bicycle 2604, the movement path and the probability of contact with user U1 are determined. Pedestrian 2703 is approaching user U1 in the Y direction, but the anticipated path k3 differs from user U1's movement path k1. In other words, pedestrian 2703 has a low priority due to the low probability of contact with user U1. Pedestrian 2702 is approaching user U1 in the Y direction, and the anticipated path k2 is close to user U1's movement path k1 (e.g., the distance in the X direction is smaller). In other words, pedestrian 2702 has a high priority because there is a possibility that it will come into contact with user U1.

[0342] Bicycle 2704 is approaching user U1 in the Y direction, and based on its relative distance and relative speed to user U1, it has a high priority. Furthermore, regarding bicycle 2704, the anticipated path k4 is close to the path k1 of user U1's movement. In other words, bicycle 2704 has a high priority because there is a possibility of it coming into contact with user U1.

[0343] Additionally, device 1 or server device 2 assigns a priority order by comparing the pedestrian 2702 and bicycle 2704, for example, by comparing their relative speeds. For example, if the relative speed of bicycle 2704 is greater than that of pedestrian 2702, bicycle 2704 becomes the first priority and pedestrian 2702 becomes the second priority. Based on these priorities, device 1 performs audio readings at designated times, for example, in the order of bicycle 2704, pedestrian 2702, and car 2701. For example, the audio output might be as follows: first, "The bicycle is approaching from behind"; second, "The pedestrian is approaching from the front"; and third, "The car has stopped in front".

[0344] By reading aloud in this way, prioritizing factors such as proximity and path, the safety of the user U1 and the surrounding environment can be improved.

[0345] [Control measures corresponding to the degree of hazard of the risk factors, and guidance of safe spaces]

[0346] Figure 28This document illustrates control examples corresponding to the degree of hazard of a hazard factor, as well as control examples for guiding a safe space. Device 1 and server device 2 can either define a predetermined object as a hazard factor, or determine an object as a hazard factor considering factors such as relative distance and speed. Here, a hazard factor refers to an object that has the potential to come into contact with user U1 and poses a potential danger to user U1.

[0347] As an example of pre-setting a predetermined object as a hazard factor, for instance, it is also possible to... Figure 10 Lane 903 and vehicle 1003 traveling in lane 903 are designated as hazard factors. In other examples, tools that pose a risk of injury or death may also be designated as hazard factors.

[0348] In the example of Figure 25, suppose user U1 is walking forward (in the +Y direction) on the left side of road 2800. Device 1 or server device 2, based on the image from camera 403 and sensors, imagines / calculates the path k10 of user U1's movement / travel direction. Suppose a bicycle 2801 and a pedestrian 2802 are detected as objects in front of user U1. Bicycle 2801 is moving towards user U1 in the -Y direction. Pedestrian 2802 is walking in the same direction as user U1 in the +Y direction. Device 1 or server device 2 imagines / calculates the path k11 of bicycle 2801. It is determined that the path k11 of bicycle 2801 is close to the path k10 of user U1, and in this state, there is a possibility of contact.

[0349] Based on this determination, device 1 or server device 2 identifies / sets bicycle 2801 as a hazard. Furthermore, device 1 or server device 2 can also determine / set the degree of hazard regarding the hazard. For example, the degree of hazard can be calculated based on the relative distance and speed between user U1 and bicycle 2801, or based on the distance between user U1's path k10 and bicycle 2801's path k11. Even if the user U1's path and the object's path have different directions, the object can still be designated as a hazard if these paths are assumed to intersect.

[0350] While the hazard level of a hazard factor can be a single value, it is not limited to this. For example, it can also be judged using multiple levels / values ​​such as high / medium / low, or 1st / 2nd / 3rd. In this example, the hazard level of bicycle 2801 is set to medium based on the relative distance D1 or relative speed V1 between user U1 and bicycle 2801. Device 1 controls the audio to read aloud about bicycle 2801, which is a hazard factor, with high priority according to the hazard level.

[0351] For example, at time point 1, bicycle 2801 is detected, and an audio reading equivalent to a basic notification such as "A bicycle is approaching from the front" is performed. Then, at time point 2, in this system, as a criterion, based on factors such as the relative speed V1 between user U1 and bicycle 2801 exceeding a threshold and the paths being close, bicycle 2801 is designated as a hazard factor, and the hazard level is set to "medium". Regarding bicycle 2801 as a hazard factor, this system sets it to high priority based on the hazard level being "medium" and determines it to be the target of the audio reading. Furthermore, based on the hazard level being "medium", this system adds a warning to the audio reading for bicycle 2801. The warning can be, for example, a human voice output such as "Please be careful" or a predetermined warning tone. The warning tone can be an example of a sound that is repeatedly emitted at a predetermined period (e.g., beep beep beep...) or an example of vibration.

[0352] Furthermore, when reading out warnings about hazards, the audio delivery / warning can be adjusted based on the level of danger. For example, the volume can be increased as the danger level rises. The timbre of the voice / warning tone can also be changed according to the level of danger. Additionally, the period of the warning sound and vibration can be shortened as the danger level increases.

[0353] Furthermore, this system has the function of using voice prompts to guide user U1 to a safe space in order to prevent / avoid danger from hazardous factors. Using the same... Figure 28 The following explanation is provided. Device 1 and server device 2, based on user U1's path k10, anticipate the bicycle 2801's path k11 until the bicycle 2801 approaches user U1 within a certain distance D11 along path k10, and then perform normal control. For example, at time point 1, the voice announces "The bicycle is approaching from the front." In normal control, the position of user U1 and path k10 are essentially maintained unchanged, anticipating and waiting for the bicycle 2801 to change its path k11 in consideration of user U1.

[0354] If bicycle 2801 enters the predetermined distance D11 on path k10 without changing path k11, the system determines that there is a possibility of contact and performs control related to guiding the safe space. Based on the image analysis of camera 403 (e.g., front camera), the system calculates the safe space and the direction of movement (in other words, the avoidance direction) to avoid contact with bicycle 2801 moving towards user U1 on path k10 on user U1.

[0355] In the illustrated example, the system first determines whether it is permissible to move left (-X) or right (+X) from the current position on road 2800. In this example, it is assumed that there is no space to the left (e.g., a wall), and there is free space 2810 to the right (in other words, space without any objects). The system determines that it is not permissible to move left, but is permissible to move right. Furthermore, the system also determines, for example, the distance to surrounding objects regarding the free space 2810 on the right. Specifically, the system compares the free space 2810 with the path k11 of bicycle 2801 and the paths of other objects, determining whether these paths intersect with the free space 2810. Based on this determination, the system considers the free space 2810 safe; in other words, it sets the free space 2810 as a safe space.

[0356] Device 1 uses the free space 2810 to the right (+X) of user U1 as a safe space to guide user U1 to move to the right. For example, as a voice reading at time point 3, corresponding to this guidance, it outputs voices such as "Please move to the right" and "Please move to the right".

[0357] As described above, the safety space also considers the path of objects approaching user U1 and selects a space that avoids such objects. This prevents / avoids contact between user U1 and objects such as bicycle 2801.

[0358] [Controlling the reading of specific objects by prioritizing them]

[0359] Figure 29 This illustrates a control example that prioritizes specific objects for audio reading. In this system, the objects to be read aloud can also be pre-defined as specific objects. This setting can be a system or app setting, or a setting for each user. For example, specific objects related to assisting visually impaired individuals, traffic safety / accident prevention, etc., can be set as specific objects.

[0360] In the illustrated example, user U1 is walking forward (+Y) on a pedestrian crossing 2900 with a lane on the right. For user U1, the following objects are detected ahead: traffic light 2901, pedestrian crossing 2902, pedestrian 2903, crosswalker 2904, and road sign 2905. This system assumes that traffic lights, pedestrian crossings, and road signs are pre-defined as specific objects. From the camera's image, this system prioritizes detecting traffic light 2901 and pedestrian crossing 2902 as specific objects, and prioritizes reading aloud the detected traffic light 2901 and pedestrian crossing 2902.

[0361] For example, when device 1 initially detects a pedestrian crossing 2902, it reads aloud, "There is a pedestrian crossing on the right." Next, when device 1 initially detects a traffic light 2901 (e.g., a green light), it reads aloud, "The signal is green." Furthermore, the system detects changes in specific objects and reads aloud at the time of that change. For example, it detects a change in the state of traffic light 2901, such as a change from green to red. When a signal changes, device 1 reads aloud the meaning of that signal change. For example, it outputs, "The signal has turned red."

[0362] In addition, this system also prioritizes traffic safety, calculating the safety / danger level of user U1 crossing pedestrian crossing 2902. For example, if traffic light 2901 is red, the system can determine a high level of danger and output a warning accordingly. For instance, it might output, "The signal has turned red. Please be careful."

[0363] Similarly, other specific objects, such as road sign 2904, can also be read aloud with high priority. In the illustrated example, when road sign 2904 indicating no pedestrian crossing is detected, phrases like "No pedestrian crossing" are read aloud first. Other examples of specific objects include intersections and tactile paving for the visually impaired.

[0364] As mentioned above, by prioritizing the reading of specific objects, efficient assistance can be achieved in scenarios such as assisting visually impaired individuals.

[0365] [Example of control using a 360-degree camera]

[0366] Figure 30 This illustrates a control example where a 360-degree camera (in other words, a panoramic camera) is used as camera 403. The illustrated image 3001 is a schematic diagram of an image obtained by the 360-degree camera. In image 3001, the center point of the circular / annular region corresponds to the position of user U1, i.e., the position of the 360-degree camera. The circumferential direction of the circular / annular region corresponds to the forward, backward, left, and right directions (in other words, azimuth). The radial direction of the circular / annular region corresponds to the vertical direction (in other words, pitch). Objects can be detected within the 360-degree panoramic image corresponding to such circular / annular regions. For example, if there is a pedestrian diagonally in front of user U1 to the left, a pedestrian image g1 is captured within image 3001.

[0367] For example, in Figure 6BUser U1 can also have a 360-degree camera with a vertically upward optical axis, such as in the hat 1D. Alternatively, a 360-degree camera with a forward-facing optical axis can be installed in front of the hat 1D, and a 360-degree camera with a backward-facing optical axis can be installed behind it. The image 3001 can be obtained using a single 360-degree camera, or it can be obtained by combining multiple cameras. Furthermore, it can also be obtained using... Figure 30 The structure of the panoramic image 3002 shown at the bottom.

[0368] [Example of a control scheme that changes sound output based on camera direction]

[0369] Figure 31 This illustration demonstrates a control example of changing the sound output (e.g., timbre) of a voice reading in front and rear cameras facing the user U1. For example, consider an object 3101 detected from the space in front of the user U1 based on the image of the front camera C1 of device 1 (smartphone 1A), and an object 3102 detected from the space behind the user U1 based on the image of the rear camera C2. Device 1 changes the sound output in a distinguishable manner for the voice reading of the object 3101 based on the image of the front camera C1 and the voice reading of the object 3102 based on the image of the rear camera C2. For example, the timbre and volume of the voice can also be changed. In the illustrated example, the control is to output a male voice (first timbre, lower voice) for the voice reading of the object 3101 in front, and a female voice (second timbre, higher voice) for the voice reading of the object 3102 in rear.

[0370] Therefore, user U1 can easily identify the position of front and back. In addition, under this control, even if multiple voices of multiple objects are read almost simultaneously, it is easy to distinguish them by the difference in timbre.

[0371] [Example of control using 3D sound output]

[0372] Figure 32 This illustrates a control example using three-dimensional sound output. In the illustrated example, user U1 is wearing smart glasses 1C and is near an intersection. There are objects 3201 detected from the front (+Y) and objects 3202 detected from the right (+X) relative to user U1. In this control example, device 1 includes a sound output device with three-dimensional sound output functionality. This three-dimensional sound output function is a function that sets a sound source in three-dimensional space and outputs sound from a speaker or similar device in a manner that sounds as if the sound were originating from that source.

[0373] In the illustrated example, device 1 determines the position L1 of the object 3201 detected in front of user U1 and the position L2 of the object 3202 detected to the right, based on the aforementioned distance determination. These positions may be illustrative. Furthermore, device 1 controls the three-dimensional sound output so that the sound about the object 3201 sounds like it originates from position L1 in front, and so on, for the sound about the object 3202, it controls the three-dimensional sound output so that the sound originates from position L2 in the right. Thus, user U1 can easily identify the direction and position of the objects. Moreover, under this control, even if multiple sounds about multiple objects are heard almost simultaneously, identification can be achieved through the three-dimensional sound output.

[0374] In addition, the following reference information can be cited as a technical example related to the three-dimensional sound output function.

[0375] [Reference Information]

[0376] SRS-RA3000 Active Speaker / Neckband Speaker Sony (sony.jp)

[0377] [Effects, etc.]

[0378] As described above, according to Embodiment 1, the image reading function can provide more appropriate assistance to users, whether visually impaired or not. For example, it can improve the safety of visually impaired people when walking and assist in the use of transportation systems, facilities, etc. Even in the case of non-visually impaired people, it can, for example, provide safety measures / crime prevention on night roads, improve safety when using a mobile phone while walking, and enhance safety when driving.

[0379] <Implementation Method 2>

[0380] use Figure 33 The accompanying figures illustrate the response output system (in other words, the AI ​​conversation system) of Embodiment 2. The basic structure of Embodiment 2 is the same as / generalized as that of Embodiment 1; the structural parts in Embodiment 2 that differ from those in Embodiment 1 are described below.

[0381] The following can be cited as examples of research topics. Previously, there were AIs that generated conversations (in other words, chat) between the user and the content input into the response input / output interface (in other words, conversational AI / chat AI, etc.). However, when using conversational AI in situations such as a user walking on the street, the surrounding environment was not considered. Even when the user takes a picture of the surrounding environment with a camera and inputs it into the conversational AI, the filming process is still required, which is cumbersome.

[0382] Therefore, in Implementation 2, when using conversational AI in situations such as a user walking on the street, multiple cameras on the device automatically capture images at various time intervals, inputting multiple images containing the surrounding environment into the conversational AI. The conversational AI generates / outputs responsive conversational statements in the following manner: along with the conversational statements input by the user, it inputs multiple images, identifies the surrounding environment from the multiple images, and reflects the identified surrounding environment in the topic. Thus, in Implementation 2, without complicated operations, the surrounding environment is reflected in the conversation with the AI, achieving an appropriate conversation.

[0383] In Embodiment 1 described above, a system was shown that assists a user's walking, etc., based on input from multiple camera images of Device 1. Even in this Embodiment 2, the point of using multiple camera images as input is common, so there are many overlapping components with Embodiment 1. Hereinafter, the description will focus on the additional components added in Embodiment 2. As a structure common to both the functions in Embodiment 2 and Embodiment 1, there is a point where time-series images captured by multiple cameras of Device 1 are used as input, the AI ​​analyzes them, identifies the user's surroundings, and generates / outputs response text matching those surroundings. As structures where the functions in Embodiment 2 differ from those in Embodiment 1, there are points where the user and AI can have a conversation, and where the AI ​​generates conversational statements while using text and images. The AI ​​conversation function, which is a function described in Embodiment 2, can be added to the voice reading function, which is a function in Embodiment 1, and both can be used together. That is, by using the function of Embodiment 1 to read aloud the surroundings, the user's walking can be assisted; simultaneously, by using the function of Embodiment 2 to reflect the surroundings, the user and AI can have a conversation.

[0384] In Embodiment 2, a function for facilitating conversations between the user and the AI ​​(sometimes referred to as an AI conversation function) is included. This conversation can also be installed as a user interface within a pre-defined application. In Embodiment 2, the user's device 1 has an interface for engaging in conversations with the AI ​​(sometimes referred to as a conversational AI) using text / voice / images. This interface includes: an interface for inputting / sending requests (in other words, instructions, questions, prompts, etc.) to the AI ​​in text / voice / images; and an interface for receiving / outputting responses (in other words, answers, etc.) from the AI ​​in text / voice / images. The conversational AI is implemented, for example, through an AI server 2 (specifically, an MLLM), but can also be implemented as a function within device 1 (specifically, a local LLM). Furthermore, the forward slash ( / ) in "text / voice / image" represents OR. That is, this representation indicates more than one of text, voice, and image.

[0385] In Implementation 2, the AI ​​conversation function sends / inputs data from multiple images (sometimes recorded as camera images) captured by multiple cameras of device 1, along with text / sound / images based on user-input conversation statements, as a request from device 1 to the conversation AI of AI server 2. The conversation AI of AI server 2 generates text / sound / image responses to the conversation statements based on the input text / sound / images. At this time, the conversation AI of AI server 2 identifies the user and the surrounding environment of device 1 by performing image recognition processing on the input multiple images (camera images). Furthermore, the conversation AI of AI server 2 controls the reflection of the identified surrounding environment in the response conversation statements. In other words, the user's device 1 is controlled to obtain conversation statements reflecting the surrounding environment from the conversation AI. For example, the conversation AI detects surrounding targets (in other words, objects) and reflects the detected surrounding targets as topics in the conversation flow. For example, the conversation AI generates conversation statements with surrounding targets as topics and inserts / appends them to the conversation flow. Examples of surrounding targets can be facilities, road signs, etc., already defined on a map, or objects not defined on a map (e.g., moving objects). This allows users to engage in conversations with AI that reflect the surrounding environment.

[0386] Furthermore, in the AI ​​conversation function of Implementation Method 2, the user's face can also be included in the camera image. The conversational AI performs image recognition processing on the input camera image to recognize not only the surrounding environment (surrounding objects other than the user's face), but also the user's facial expressions and actions (e.g., facial orientation, movement, gaze direction, etc.). Expressions and emotions have a corresponding relationship. Known techniques can be used to classify expressions / emotions. For example, categories such as joy / anger / sorrow / happiness / no emotion can be given.

[0387] Furthermore, the conversational AI control incorporates the identified facial expressions and actions along with the identified surrounding circumstances into the corresponding conversational statements. In other words, the user's device control retrieves conversational statements from the conversational AI that reflect the user's facial expressions and actions. For example, the conversational AI generates conversational statements that use the user's facial expressions and actions as topics and inserts / adds them to the conversation flow. Alternatively, the conversational AI can also control the content of subsequent conversational statements based on the user's facial expressions and actions. For example, when providing guidance related to the surrounding circumstances, such as navigation on a map, the conversational AI can modify or add guiding conversational statements based on the user's facial expressions and actions. For example, the conversational AI can also control whether to continue / expand the topic if the user displays a happy expression regarding a topic related to a surrounding target, or to stop the topic or change to another topic if the user displays an uneasy or sad expression regarding the topic. Additionally, for example, the conversational AI can also control whether to add guiding conversational statements to alleviate the user's uneasy or sad expressions when providing route navigation to a destination on a map.

[0388] [System Architecture]

[0389] Figure 33 The system structure of Embodiment 2 is shown. The system structure of Embodiment 2 is similar to... Figure 1 The system structures are largely the same. The system in Implementation Method 2 (… Figure 33 This is an example of an AI chatbot service (in other words, an AI conversational service) applied to communication network 9. The operator provides this service to each user's device 1 through AI server 2. Figure 33 and Figure 1 The difference lies in that device 1 is an AI conversation device corresponding to the AI ​​conversation function. Additionally, in AI server 2, as AI 20, there is a conversation AI 40 corresponding to the AI ​​conversation function. Conversation AI 40 generates a response conversation statement in response to the conversation statement input from device 1 and outputs it to device 1. This system has an AI conversation function, which is the function of engaging in conversation with conversation AI 40 while using images captured by multiple cameras on device 1 set up on user U1. Embodiment 2 can also apply... Figure 2 , Figure 3 The structure.

[0390] As for device 1, similarly to embodiment 1, various devices can be applied. Similarly, the devices in embodiment 2 can also be applied in the same way. Figure 5A , Figure 5B , Figure 6A ,as well as Figure 6B Various devices 1 described herein. For example, a smartphone 1A ( Figure 6A As multiple cameras, it is possible to use the front camera C2 (inner camera) and the rear camera C1 (outer camera). User U1 communicates with the conversation AI40 through images captured by the front camera C2 (inner camera).

[0391] The conversational statements created by user U1 through text and voice input to smartphone 1A are input into conversation AI 40, and multiple images captured by multiple cameras of smartphone 1A are also input into conversation AI 40. These images include those captured by rear camera C1 (external camera) showing the surrounding environment (e.g., the actual view in front) and those captured by front camera C2 (internal camera) showing user U1's face. During the conversation with the AI, images from each camera are continuously input along a timeline. Furthermore, the input is not limited to using rear camera C1 (external camera); cameras from other shooting directions (e.g., left and right cameras) can also be used.

[0392] The conversational AI 40 generates corresponding conversational statements (sometimes referred to as output conversational statements) based on text / sound / images (sometimes described as input conversational statements) input from the user U1's device 1, and outputs them to the device 1. At this time, the conversational AI 40 utilizes input image information from multiple cameras. The conversational AI 40 controls the recognition of surrounding conditions, facial expressions, and actions from the camera images and reflects these in the corresponding conversational statements. The user U1's device 1 outputs the conversational statements from the conversational AI 40 in a predetermined format, such as text / sound.

[0393] For example, Conversation AI40 generates conversational statements that use identified surrounding targets as topics and inserts them into the conversation flow. Additionally, for example, Conversation AI40 generates conversational statements that use facial expressions as topics and inserts them into the conversation flow. Conversation AI40 can also refer to user U1's facial expressions and gestures to study / modify the way conversational statements are responded to.

[0394] In a conversation, the conversational AI40, based on camera images as the surrounding environment and nearby targets (such as a bicycle approaching from in front of user U1), generates conversational statements such as "A bicycle is approaching from the front," urging the user to pay attention to their surroundings, and inserts these statements into the conversation flow. Additionally, when a famous landmark is identified as a surrounding environment, the conversational AI40 generates conversational statements using that landmark as a topic, such as "The building seen to the right is the famous Roppongi Hills," and inserts these statements into the conversation flow. In this way, the conversational AI40 generates conversational topics based on camera image recognition.

[0395] In addition to images of surrounding targets captured by the external camera and images of the face captured by the internal camera, the conversational AI 40 can also utilize detection information of the device 1's posture (vertical and horizontal, tilted state), position, orientation, and the user U1's gaze (gaze direction, gaze point, etc.). For example, the conversational AI 40 can also generate conversational statements by prioritizing surrounding targets in front of the device 1 and surrounding targets in front of the user U1's gaze as topics.

[0396] In this second embodiment, not only are multiple camera image inputs described in embodiment 1 used, but also text input by the user, such as voice or text from voice recognition, and text input from buttons, etc., are used as input information to the conversation AI. The conversation AI can generate corresponding conversation text / voice based on the input conversation text / voice.

[0397] Furthermore, techniques for AI chat based on text input are known. For example, Japanese Patent No. 6433614 discloses a system that uses log information and chatbot evaluation information to more appropriately implement chat in a conventional chatbot system.

[0398] The AI ​​chat-related technical aspects of this embodiment 2 (particularly the conversational AI 40) can utilize AI technologies for chatbots, as is commonly known. Alternatively, a dedicated AI for this embodiment can be installed.

[0399] [Structure of equipment and servers]

[0400] Figure 34A This shows a structural example of device 1 in embodiment 2, and... Figure 7A The difference lies in the fact that the non-volatile memory 303 contains AI chat software 347 (in other words, an AI conversational application). Furthermore, the AI ​​data 343 includes not only data related to the image reading software 342, but also data related to the AI ​​chat software 347.

[0401] Figure 34B This illustrates a structural example of server 2 in embodiment 2, as a comparison with... Figure 7B The difference lies in the fact that the non-volatile memory 403 contains AI chat software 447 (in other words, an AI conversation server). Furthermore, the AI ​​data 443 includes not only data related to the AI ​​software 442, but also data related to the AI ​​chat software 447.

[0402] [Processing Flow: AI Conversation Method]

[0403] The processing flow in Implementation Method 2 is as follows: Figure 4Different processes. The same principle can be applied in implementation method 2. Figures 8-32 The structure shown includes settings, image acquisition, image parsing, image reading data creation, and image reading.

[0404] Figure 35 The processing flow corresponding to the method in Implementation 2 (AI chat method, AI conversation method, etc.) is shown. Figure 33 Execute in the system shown above Figure 35 The process. That is, the processor 301 of device 1 and the processor 401 of server 2 follow... Figure 35 The process execution and processing.

[0405] In step S201, when user U1's device 1 receives an instruction from user U1 and the device 1's state meets specific conditions, the AI ​​conversation function is enabled (activated).

[0406] As the actual state of AI20 (Session AI40) and AI Function 20b, it is for the purpose of... Figure 34A , Figure 34B The processors 301 and 401 in the structure shown read the AI ​​chat software 347 on the device 1 side and the AI ​​chat software 447 on the server 2 side, either individually or together with the image reading software 342 and the image reading software 442, and execute the processing state (in other words, execute the module or instance).

[0407] AI20 (Special Conversational AI40) consists of generative AI, LLM (Large-Scale Language Model), etc. The LLM can also be a multimodal LLM (MLLM). Furthermore, in this specification, the actual state of the AI ​​is sometimes referred to as the AI ​​process circuit.

[0408] In step S202, device 1 receives voice input or text input from user U1. In the case of voice input, device 1 can also perform voice recognition on voice input from a microphone or other source and convert it into text. Device 1 sends the voice or the text of the voice recognition result, or the text of the text input result, to AI 20 or AI function 20b based on an instruction (operation input) from user U1 or after a predetermined time has elapsed since the input.

[0409] Additionally, in step S203, device 1 captures / acquires multiple images by taking images from input images of multiple cameras with different shooting ranges / directions at predetermined time intervals, either at the same time as in step S202 or at close time intervals with a predetermined relationship. AI function 20b then acquires the captured camera images (multiple images, image information). This is achieved through the implementation method 1 described above (…). Figure 8 As shown in the figure, the shooting timing of multiple cameras is controlled according to the settings to capture the image at that time.

[0410] For example, in smartphones such as 1A ( Figure 6A , Figure 9A In a scenario where there is an inner camera C2 (located on the front side of the frame, shooting from the rear) and an outer camera C1 (located on the back side of the frame, shooting from the front), during an AI session, image input including a facial image from the inner camera is performed, and at a predetermined time, image input including a frontal image from the outer camera is also performed. Similar to Embodiment 1, the processing ratio (e.g., ...) can be set between the image input from camera C1 and the image input from camera C2. Figure 9A State A). The same applies when using cameras in the front, back, left, and right.

[0411] AI20 or AI function 20b determines and outputs the content of the response dialogue statement based on the user U1's input text or voice and camera images (multiple images). At this time, AI20 or AI function 20b controls the use of images of the surrounding environment captured by an external camera, or images of the surrounding environment excluding the face captured by an internal camera, to reflect the identified surrounding environment in the response dialogue statement. Alternatively, AI20 or AI function 20b can also be controlled to analyze the user U1's facial expressions and gestures from the facial images of the internal camera and reflect the analyzed expressions in the response dialogue statement.

[0412] Additionally, AI20 or AI function 20b can also detect the posture (horizontal, horizontal, tilted, etc.), position, direction, facial orientation, activity, and gaze of device 1. This system can also generate conversational statements by inferring, for example, nearby targets close to user U1's line of sight based on the detected information and introducing those targets into the conversation.

[0413] Furthermore, AI20 or AI function 20b can also analyze facial images to determine whether user U1 is looking at the display screen in front of device 1, looking in a certain direction, or whether their gaze is frequently shifting or stable and focused. When user U1 is holding device 1, the movement of their hand, the position of the device 1, and their posture are also monitored in real time. These changes can be detected based on the sensors on device 1 (such as posture detection unit 312) and the content of the camera image.

[0414] The AI ​​function 20b (AI conversation application, in other words, client software) on device 1 can also perform preprocessing on device 1, that is, sub-processing before the main processing by AI 20 on server 2. AI function 20b can perform text shaping on conversational statements, as well as transformations such as audio-to-text and image-to-text conversions. Furthermore, AI function 20b can extract the parts and areas of focus from the camera image through the user U1's line of sight, and can also perform image processing such as applying edits related to the areas of focus to the camera image.

[0415] In step S204, device 1 sends the input conversation statement, camera images (multiple images), and preprocessing information as a communication request (data according to a predetermined communication protocol) to AI 20 on server 2. Alternatively, device 1 can also send the retrieved information (input conversation statement and camera images) to AI 20 on server 2 without performing any action in AI function 20b. Furthermore, it can also be used in conjunction with embodiment 1 (… Figure 3 Similarly, a series of processes can be completed only on the device 1 side. In this case, device 1 can process the data through AI function 20b without sending information to server 2.

[0416] In step S205, the server 2 of this system uses AI20 (session AI40) to parse the text of the input session statement and the camera image obtained from device 1, and further parses the preprocessed information if available. Specifically, in step S205, AI20 parses / identifies the surrounding situation from the camera image (e.g., an image from an external camera), and as a result, obtains text representing the surrounding situation.

[0417] Alternatively, step S206 can be added. In step S206, AI20 parses / identifies the user's facial expressions and actions from the camera image (e.g., an image from an internal camera), and as a result, obtains information representing the expressions and actions.

[0418] In step S207, AI20 generates at least one candidate response conversation statement (text / sound / image) based on the results of steps S205 and S206, namely the parsing results of the input conversation statement and the camera image (surrounding conditions, expressions, etc.). Alternatively, in step S207, no candidates may be generated, and only one response statement may be determined from the beginning. Typically, the response conversation statement (output conversation statement) is text generated using text representing the surrounding conditions and expressions as recognition results (in other words, a statement composed of strings, etc.). The response conversation statement may also replace text, or be generated / output together with text, based on sound or images (still images, videos).

[0419] In step S208, the system (e.g., AI20 of server 2) selects one response statement candidate as a result of step S207 based on a predetermined judgment (e.g., the judgment of the degree of change described in embodiment 1) and a timing, thereby determining the session statement to be output (output session statement) and sending the information to device 1.

[0420] In step S209, device 1 receives the output session statement information from AI20 determined in step S208, and automatically outputs the output session statement to user U1 in a predetermined output format. This output can be a reading aloud (in other words, audio output), a display of text on a screen, or a display of an image. The output format is configurable.

[0421] In step S210, device 1 confirms whether the AI ​​session function is set to the off state. If it is set to the off state ("Yes"), the process ends. If it remains on state ("No"), it returns to step S201 and repeats the same process.

[0422] As described above, in Embodiment 2, in a system having a conversational AI that responds to conversational statements formed by user U1 through voice or text input, the response made by the conversational AI can be optimized by inputting information from one or more camera images at appropriate timings. This enables the realization of appropriate conversations that reflect the user U1's surroundings, expressions, etc.

[0423] [System Architecture Example: Software]

[0424] Figure 36A Examples of system structures in Implementation 2 are shown, particularly examples of structures related to software.

[0425] Figure 36A Systems (in other words, AI conversational systems, etc.) and Figure 33 Similarly, the system is a user terminal (device 1) acting as user U1 and a server (server 2) acting as AI server, which are appropriately connected via communication. Device 1 includes an APP 351 as a client application for implementing AI conversation functions. Server 2 includes an AI 352 as server software for implementing AI conversation functions. Furthermore, APP 351 includes a conversation client 353 and an image recognition client 354 as components of programs, etc. AI 352 includes a conversation AI 355 and an image recognition AI 356 as components of programs, etc.

[0426] exist Figure 36AIn the system, the session client 353 creates an input session statement (instruction statement, prompt word) based on the user's text / voice input and sends it as a request a1 (e.g., a packet on a communication network) to the server 2. Additionally, the image recognition client 354 acquires camera images (multiple images) and sends the data of the camera images (multiple images) a2 to the server 2 at roughly the same timing as the input session statement (request a1). The data of request a1 and camera image a2 can either be sent together at approximately the same timing or sent separately at different timings.

[0427] Server 2's session AI 355, based on the input session statement of request a1 and the session generation model, generates a response session statement (output session statement). At this time, Server 2 also uses image recognition AI 356. Image recognition AI 356 takes the camera image a2 as input, and based on image recognition processing, identifies the surrounding situation or facial expressions and actions, obtaining information a3 such as text representing the surrounding situation or text representing facial expressions and actions. Session AI 355 inputs this information a3 along with the input session statement, and based on this information, generates an output session statement that reflects the surrounding situation or facial expressions as topics. Session AI 355 sends this output session statement (text / sound) as a response a4 to device 1. Device 1's session client 353 outputs this output session statement to user U1 using text / sound / image.

[0428] Regarding the series of processes related to the aforementioned AI conversational function, as a typical example, user U1 can input conversational statements by voice and output conversational statements in response from the AI ​​by voice. As another example, user U1 can input the text of conversational statements on the screen, and the text of the conversational statements in response from the AI ​​can be displayed on the screen.

[0429] exist Figure 36A In this system, as a variation, the image recognition AI 356 can also be integrated into the conversation AI 355. Alternatively, the conversation client 353 and the image recognition client 354 can also be integrated.

[0430] Figure 36B The system is relatively Figure 36AThe main difference lies in the addition of an image recognition AI 357 on device 1. This AI 357 utilizes a local LLM implementation. It takes the camera image from device 1 as input, processes it based on image recognition, identifies the surrounding environment or facial expressions and gestures, and obtains text representing the surrounding environment or facial expressions and gestures. At roughly the same timing as the input session statement (request a1), device 1's image recognition AI 357 sends this information a3 to server 2.

[0431] Server 2's session AI 355 receives input session statements and surrounding information (such as facial expressions) from input request a1 (a3). Based on the session generation model, it generates session statements (output session statements) that reflect the surrounding situation or facial expressions as topics. Session AI 355 sends the text / sound of this output session statement as a response (a4) to device 1. Device 1's session client 353 then outputs this output session statement to user U1 using text / sound / image.

[0432] Regarding the aforementioned image recognition AI356 and 357, in particular, well-known object detection techniques such as CNN (Convolutional Neural Network) can be applied.

[0433] In the processing of text / sound statements and camera images in the aforementioned conversations, time differences sometimes arise due to computer processing time. However, these processes can be synchronized as much as possible by setting / controlling the timing of various data / information transmissions. For example, in Figure 36A In the process, after the image recognition processing by AI356 is completed, timing alignment can be performed to input the input session statement (request a1) and information (a3) ​​to the session AI355, generating a response. Figure 36B In addition, after the image recognition processing of AI357 is completed, the timing can be aligned and the request a1 and information a3 of the session statement can be sent.

[0434] As another variation, the image recognition AI used to identify the surrounding environment (surrounding targets, etc.) and the image recognition AI used to identify facial expressions and movements can be set up separately. Additionally, as another variation, the image recognition AI can be set up for each of the multiple cameras in device 1. For example, when using two cameras—an internal camera and an external camera—in smartphone 1A, the image recognition AI for the internal camera images and the image recognition AI for the external camera images can be set up separately.

[0435] Figure 36C The system is relatively Figure 36AThe main difference lies in the implementation: Device 1 has a conversational AI 358, while Server 2 has an image recognition AI 356. The conversational AI 358 on Device 1 utilizes a local LLM implementation. Device 1's image recognition client 354 sends a camera image a2 to Server 2. Server 2's image recognition AI 356, based on the camera image a2 and image recognition processing, obtains information a3 representing the surrounding environment, facial expressions, etc. Server 2's image recognition AI 356 sends this information a3 as a response to Device 1. The image recognition client 354 then transmits this information a3 to the conversational AI 358. The conversational AI 358 takes the input conversational statement from the user and the information a3 as input, and, based on the conversation generation model, generates a conversational statement (output conversational statement) that reflects the surrounding environment, facial expressions, etc., as topics. The conversational AI 358 outputs this output conversational statement to user U1 using text / sound / image.

[0436] Figure 36D The system is relatively Figure 36A The changes, for the sake of Figure 3 Similarly incorporated into the structure of device 1, the main difference is that device 1 includes conversation AI 358 and image recognition AI 357. On the server 2 side, the conversation AI and image recognition AI of embodiment 2 are not required. When used in conjunction with the voice reading function of embodiment 1, the server 2 side only becomes an installation related to that voice reading function.

[0437] Although Figure 36A As not shown in the figure, functions related to voice reading in Implementation 1 can also be installed on server 2. For example, AI 20 (the AI ​​that identifies objects from the input image and outputs them as text) in Implementation 1 and various AIs (conversation AI, image recognition AI) in Implementation 2 can be installed together.

[0438] [AI Conversation Functionality: Structural Example]

[0439] Figure 37 Showing based on Figure 36A A more detailed structural example related to AI conversational functionality. Figure 37 In the system, server 2 includes a session AI 355, which in turn includes an image recognition AI 356. Device 1, as a functional unit, includes a setting unit 371, an image acquisition unit 372, a session input unit 373, a session output unit 374, multiple cameras 375, an input device 376, and an output device 377. Each functional unit is implemented through processor processing and circuit installation.

[0440] The setting unit 371 is responsible for system and user settings related to AI conversation functions, and also provides a user interface for these settings. Multiple cameras 375 are multiple cameras with different shooting ranges / directions, such as the aforementioned internal and external cameras. Multiple images b1 (including images from each camera) are obtained from the multiple cameras 375. Figure 8 The camera input image 801 is equivalent to this.

[0441] The image acquisition unit 372 acquires captured images (and images) at predetermined time intervals based on multiple images b1 from multiple cameras 375. Figure 8 (equivalent to 801), and is transmitted to the session input unit 373 as multiple images b2 (image information).

[0442] The conversation input unit 373 acquires text / sound / image b3 (user input information) input by the user via the input device 376, and creates an input conversation statement (instruction statement, prompt word, etc.) for input to the conversation AI 355 based on the user input information. The conversation input unit 373 sends / inputs the input conversation statement and multiple images (image information) to the conversation AI 355. When the user input is in the form of sound, the conversation input unit 373 acquires sound data via the microphone, performs sound recognition processing on the sound data to convert it into text, and creates a text-based input conversation statement. Additionally, at this time, the conversation input unit 373 attaches multiple images b2 to the input conversation statement, creating a request b4 for sending to the server 2. Figure 37 Request b4 in the middle and will Figure 36A The request a1 and camera image a2 are combined into a single request. There are no restrictions on the form of attaching multiple images b2 to the input session statement (the form of request b4); as an example... Figure 38 As shown in the image.

[0443] The conversation input unit 373 sends request b4 to server 2. Server 2 receives request b4, and conversation AI 355 extracts the text of the input conversation statement and multiple images from request b4. Conversation AI 355 parses the input conversation statement and multiple images to generate text / sound / images that become the response statement (output conversation statement) for the conversation. Conversation AI 355 uses image recognition AI 356 to recognize surrounding conditions, expressions, etc. from the multiple images, obtaining text representing the surrounding conditions, expressions, etc. Conversation AI 355 takes the text of the input conversation statement and the text representing the surrounding conditions, expressions, etc. as input to generate text of the response conversation statement (output conversation statement) that reflects the surrounding conditions, expressions, etc. to the topic. In addition, at this time, conversation AI 355 can also generate a first output conversation statement for the input conversation statement and a second conversation statement for the text representing the surrounding conditions, expressions, etc., and output them in groups. Server 2 (conversation AI 355) sends the output conversation statement as response b5 to device 1.

[0444] Device 1 receives response b5, and session output unit 374 extracts the text of the output session statement from response b5 and outputs the output session statement from output device 377 in a predetermined form. When the output form is sound, session output unit 374 generates sound data through sound synthesis processing based on the text of the output session statement, and outputs sound from the speaker (output device 377) based on the sound data (b6).

[0445] When identifying the surrounding situation from camera images, this system can also use detection information such as the position / orientation / pose of device 1, as well as information based on map databases, etc. By using these together, the accuracy of the surrounding situation identification can be improved. When image recognition processing is performed by AI on server 2, the detection information and other related information can also be sent from device 1 along with the camera images.

[0446] [Request Data]

[0447] Figure 38 This shows a summary of the structure of request b4. Request b4 includes text data 3801 of the input conversation statement (prompt word) and image data 3802 of multiple images (camera images). The image data 3802 of the multiple images includes image data for each camera (corresponding shooting direction) (image data of camera 1 to camera N). For example, in... Figure 6A In the case of two cameras (C1 and C2) positioned in front and behind, the first camera image data is image data from the outer camera (C1) shooting from the front, and the second camera image data is image data from the inner camera (C2) shooting from the rear. Each camera image data also includes information such as shooting direction and shooting time. This request b4 data is sent and received as data such as packets corresponding to the communication protocol.

[0448] Furthermore, the example of request b4 illustrated illustrates a case where both the input session statement and the camera image are generated at substantially the same time. However, there are also cases where only the input session statement or only the camera image is generated at a certain point in time. In these cases, the content of request b4 is either text data 3801 or image data 3802. Additionally, the example of request b4 illustrated illustrates a case where multiple images obtained from multiple cameras at substantially the same time are sent as a group, but this is not limited to this. As described above. Figure 9A As shown, when multiple cameras are used to shoot at staggered times, at predetermined periods, or at predetermined ratios, corresponding data sets can be created. For example, in... Figure 9A In the case of control 1, the data can also be grouped as image data 3802, with each time point being a separate camera image data point, and these can be sent sequentially. As another example, camera image data from multiple close time points can be collected in a single request b4. For instance, image data 3802 can also contain four camera image data points from time points 1 to 4 of control 1.

[0449] [Map and surrounding area]

[0450] Figure 39 This illustration shows an example of the surrounding environment of user U1 and device 1 when using the AI ​​conversation function of embodiment 2. It is a schematic diagram of the current location of user U1 and device 1, roads, surrounding objects, etc., viewed from above on a map. The map has directions such as east, west, north, and south, and latitude / longitude (not shown), corresponding to a spatial coordinate system (X, Y, Z). In this example, the Y direction is north. User U1 (with smartphone 1A) of device 1... Figure 6A (Same) represents the state of being at a location L1 (current position) on the map. Location L1 is represented by location coordinates (X1, Y1, Z1), corresponding latitude / longitude / altitude, etc. At location L1, user U1 and smartphone 1A are facing north (+Y direction). Direction DC1 is the shooting direction of the external camera (camera C1) of smartphone 1A, and direction DC2 is the shooting direction of the internal camera (camera C2) of smartphone 1A. Path 3901 is an example of a movement path from user U1's current location L1 to destination 3902 (corresponding to the surrounding target SO3). In this example, user U1 has a destination 3902 and moves towards destination 3902, but in other examples, sometimes there is no specific destination.

[0451] Examples of surrounding targets include surrounding targets SO1 to SO8. In this example, surrounding targets are buildings, facilities, road signs, etc., that exist on the map. Surrounding targets are not limited to these. The surrounding targets processed in Embodiment 2 are any targets that can be identified / detected from camera images. Furthermore, surrounding targets may be pre-registered to services / databases such as maps and navigation systems, or they may not be registered. In the case of targets such as facilities registered to services such as maps, information about those facilities can also be provided through those services, but in the functionality of Embodiment 2, there are discrepancies regarding surrounding targets such as those facilities, which may be reflected in the session with the AI. Additionally, in Embodiment 2, surrounding targets not registered to maps, etc., can also be reflected in the session with the AI.

[0452] Additionally, line-of-sight direction 3903 is an example of the line-of-sight direction starting from user U1 at position L1, which is approximately 30 degrees diagonally forward and to the right relative to north (+Y direction). In the case of line-of-sight direction 3903, within user U1's field of view, nearby targets SO1 and SO2 are seen (in other words, these surrounding targets are captured in the corresponding camera image), but surrounding targets SO4, etc., are not seen. Furthermore, the field of view and line-of-sight direction are irrelevant to whether user U1 actually perceives surrounding targets SO1, etc.

[0453] [Multiple images (camera images)]

[0454] Figure 40 This example illustrates how device 1 captures multiple images (camera images) using multiple cameras and sends them to server 2. This example demonstrates how... Figure 6A The two images are captured by two cameras (C1 and C2) in front and behind. Image 4001 (A) is an image captured by the external camera (camera C1) in front of user U1, and image 4002 (B) is an image captured by the internal camera (camera C2) in the rear at approximately the same time. Figure 40 Examples and Figure 39 Examples of the corresponding situations.

[0455] In the foreground image 4001, for example, surrounding targets SO1 and SO2 are captured, and these surrounding targets can be detected based on image recognition. For example, a shop can be detected as surrounding target SO1 based on the text on a billboard. For example, a building can be detected as surrounding target SO2 based on the shape of a tower. Trees can also be identified as examples of other surrounding targets. The × mark 4003 is an example of a gaze point 4003 corresponding to the direction of the gaze.

[0456] In the rear image 4002, no particularly prominent surrounding targets were captured or detected. The user's face (corresponding facial region 4004) is captured in the rear image 4002, so facial features, expressions, and actions can be identified / detected based on the image. As for actions, the orientation and movement of the face can be determined. Furthermore, the direction of gaze can be inferred based on the state of the eyes. For example, facial expressions can be categorized into multiple types (emotion types) such as joy / anger / sorrow / happiness / no expression using known emotion recognition techniques.

[0457] In the functionality of Implementation 2, for example, surrounding targets captured within the image 4001 in front, such as surrounding targets SO1 and SO2, are identified / detected and reflected as topics in the AI's conversation. Specifically, when using gaze direction control, surrounding targets closer to the gaze direction (gaze point) among all surrounding targets in the image are given priority. For example, in image 4001, the gaze direction (corresponding gaze point 4003) is to the right, so even if some surrounding targets are detected from the left-hand area, the right-hand surrounding targets SO1 and SO2 are prioritized. The distance between the gaze point 4003 and the surrounding targets (representing position coordinates) can also be determined. A region can be divided with a predetermined radius centered on the gaze point 4003, and it can be determined whether there are surrounding targets contained within that region.

[0458] [Specific example (1)]

[0459] A specific example of utilizing the AI ​​conversation function of Implementation Method 2 is explained. Figure 41 This shows the status of user U1 on the map corresponding to this specific example. Figure 42 as well as Figure 43 Is with Figure 41 The specific example of the conversation process with the AI ​​corresponding to the situation.

[0460] exist Figure 41 In a specific example, user U1 uses a map app on smartphone 1A to receive navigation (in other words, guidance) to their destination. This map app uses or integrates conversational AI. For example, in... Figure 36A In this context, conversational AI 355 can be considered as a conversational AI that incorporates map services, and conversational client 353 can be considered as a conversational client that incorporates a web browser and other functions to receive map services. The conversational AI navigates to its destination through a conversation with user U1. During this navigation conversation, the conversational AI uses surrounding objects identified from camera images as topics.

[0461] exist Figure 41Initially, user U1 is set as the current location (current position) at location p0. User U1's destination 4101 is facility "XXX" on the map, set as SOX as a surrounding target identified from the image. Location p1, etc., are examples of changing current locations. Path 4110 is the movement path from location p0 to destination 4101, an example of a route recommended by navigation.

[0462] exist Figure 42 First, user U1 starts the APP on smartphone 1A ( Figure 36A In the context of a session client 353, for example, using voice input, "Hello. I want to go to 'XXX'." (set as input session statement 4201). Smartphone 1A sends a camera image along with this input session statement 4201 to server 2. Server 2 ( Figure 36A The conversational AI 355 (especially in map apps) responds to the input conversational statement 4201, for example, generating a conversational statement like "Hello user U1. A route has been set from the current location to the destination 'XXX'. Display map. 15 minutes walk to the destination." to the smartphone 1A (outputting conversational statement 4202). The route here is related to... Figure 41 The path 4110 corresponds to this. The smartphone 1A outputs sound to the output conversation statement 4202 and displays a map on the screen.

[0463] User U1 follows the map navigation to destination 4101. User U1 travels along the suggested route (path 4110). The conversational AI outputs navigation such as "Please go straight ahead (north) for a distance" (output response statement 4203). User U1 follows the navigation and travels straight.

[0464] User U1, for example, arrives at position p1 in front of traffic light A (intersection A). Unsure where to turn left, User U1 might ask, "Should I turn left at the next traffic light?" (input dialogue statement 4204). At this time, the user's facial expression remains normal. In response to input dialogue statement 4204, the dialogue AI might output, for example, "No. Please walk along the road to the second traffic light ahead." (output response statement 4205). User U1 then continues straight following the navigation, for example, arriving at position p2.

[0465] User U1, while continuing straight but still unsure where to turn left, remains silent and displays an anxious expression. A camera image capturing this expression is sent to server 2 (silence, camera image 4206). The conversational AI, based on this camera image, identifies / classifies User U1's expression as sadness and generates / outputs navigation such as "Almost there. The left turn signal is visible." (output response statement 4207). Following the navigation, User U1 turns left at signal B (intersection B) (silence, camera image 4208). At this point, User U1 remains silent, with a normal expression.

[0466] Furthermore, based on changes in the camera image at this time, the system can also recognize left-turn actions. Additionally, at this time, from the camera image, for example, a pedestrian 4105 at the left-turn destination can be identified / detected as a moving object within the surrounding area. In this case, the presence of the pedestrian 4105 can also be reflected in the conversation (navigation). The conversational AI can, for example, provide navigation such as, "Please cross the traffic light in the direction of the pedestrian."

[0467] Next, the conversational AI outputs navigation such as "Please go straight to 'YYY'." (output conversation statement 4209). Based on map data or the identification of surrounding targets, the conversational AI recognizes that there is a facility "YYY" on the route and can provide such navigation. User U1 follows the navigation and goes straight, for example, to location p3. User U1 then makes a sound such as "I see 'YYY'" (input conversation statement 4210). The user's expression at this time is normal.

[0468] The conversational AI uses camera images as a basis to detect objects around user U1, such as a bicycle approaching from behind (same as in implementation 1). To draw user U1's attention, the conversational AI generates / outputs a response such as "A bicycle is approaching from behind. Please be aware." (outputting conversational statement 4211).

[0469] User U1 arrives at position p4, near the right-turn signal light C (intersection C), close to the "YYY" mark. The conversational AI outputs navigation such as "Please turn right at the intersection where 'YYY' is located" (output conversation statement 4212). User U1 turns right at intersection C according to the navigation (silence, camera image 4213). At this time, the user is silent, and their expression is normal.

[0470] Figure 43 yes Figure 42Continuing. User U1, for example, arrives at location p5. The conversational AI, based on the camera image, identifies stone tablet 4103 as a surrounding target. In the conversational AI, as a response to the identified stone tablet 4103 as a topic, it outputs a response such as "The stone tablet that can be seen on the left is the one with the famous song by B engraved on it." (output conversational statement 4214). User U1 responds to the output conversational statement 4214 by, for example, uttering "That's right." (input conversational statement 4215). At this time, the expression is happy.

[0471] Furthermore, the conversational AI can either expand on topics related to the monument 4103 based on the user U1's expression when responding (inputting conversational statement 4215), or it can stop. For example, in the case of a happy expression, it can also output conversational statements that expand on the same topic.

[0472] The AI ​​continues navigation. User U1 arrives at location p6. Based on map data or the identification of surrounding targets, AI identifies the traffic light D (intersection D) where a right turn is required. For example, AI identifies church 4104 at traffic light D where a right turn is required. AI outputs navigation such as "You will see a church with a cross diagonally ahead to the right. Please turn right at the traffic light immediately ahead there." (output dialogue statement 4216). User U1, for example, says "OK" and turns right at traffic light D (input dialogue statement 4217). The user's expression is normal at this point.

[0473] User U1 travels straight along the route after turning right, arriving at location p7. The conversational AI, from the camera image, identifies facility "XXX," which is destination 4101, as a surrounding target SOX. For example, suppose a part of a tower that constitutes facility "XXX" is detected diagonally to the left in the camera image. The conversational AI reflects this surrounding target, for example, outputting navigation such as "The tower of destination 'XXX' is visible diagonally to the left" (output conversation statement 4218). User U1 responds to output conversation statement 4218 by uttering, for example, "Is that it?" (input conversation statement 4219). At this point, user U1's line of sight is diagonally to the left, consistent with the direction of facility "XXX."

[0474] Based on the camera image and the user's gaze direction, the conversational AI determines that user U1 has seen facility "XXX" and outputs navigation such as "Yes. Entrance is 20m ahead." (output conversation statement 4220). User U1 follows the navigation to destination 4101 "XXX". User U1 then says something like "Arrived!" (input conversation statement 4221). The user's expression is happy at this point. Based on arrival at destination 4101, the conversational AI outputs something like "Arrived. End route guidance." (output conversation statement 4222), ending the route navigation.

[0475] Based on the specific example above, by reflecting the surrounding conditions of user U1 into the AI's conversation, more appropriate conversations, such as route navigation and guidance, can be achieved.

[0476] [Specific example (2)]

[0477] Provide other specific examples. Figure 44 Examples of conversations with AI in other scenarios are shown. User U1's scenario is that they have just arrived at Tokyo Station, have an appointment to meet someone, but still have time before that appointment, and are not limited to a single purpose. User U1, in response to the app on device 1, enters, for example, "Hello. I've arrived at Tokyo Station today" (input conversation statement 4401). From the input conversation statement 4401 and the camera image, the conversation AI identifies / infers that User U1 is at Tokyo Station, specifically at the Yaesu Central Exit. The conversation AI responds in a manner reflecting the identification result (specifically, "Yaesu Central Exit"), for example, "Hello. It seems you're at the Yaesu Central Exit of Tokyo Station. Is there anything I can help you with?" (output conversation statement 4402).

[0478] User U1 outputs conversation statement 4402, for example, "Although I have a meeting scheduled at 'AAA,' I still have 1 hour to spare." (Input conversation statement 4403). Based on the input conversation statement 4403 (information like "AAA"), the conversation AI can correctly deduce that the user is at Tokyo Station (Special Yaesu Central Exit). Furthermore, based on the input conversation statement 4403, the conversation AI can also determine the user U1's concerns, such as the destination "AAA" on the map (e.g., a specific facility) and the 1-hour reservation.

[0479] The conversational AI, in response to input conversation statement 4403, generates topics related to the information it possesses, such as outputting a response like "To get to 'AAA,' it's faster to go left past 'BBB.' There are also specialty shops to the left." (output conversation statement 4404). In this example, the conversational AI suggests routes to the destination "AAA," facilities along the route, etc.

[0480] User U1 responds to output dialogue statement 4404, for example, by saying, "I can buy local specialties when I come back. Let's just grab something to eat." (Input dialogue statement 4405). In this example, user U1 reacts negatively to the topic (local specialties) suggested by the dialogue AI, but instead shows more interest in dining. Based on this response (input dialogue statement 4405), the dialogue AI can understand that user U1's concerns are negative about local specialties and positive about dining.

[0481] After entering conversation statement 4405, user U1 walks in a certain direction within Tokyo Station. The conversational AI identifies surrounding targets from the current camera image to determine user U1's direction and location. The conversational AI generates topics corresponding to the identified direction and location. For example, the conversational AI outputs a response like, "There's a newly opened 'DDD' in the direction of 'CCC' that I'm heading. They seem to like sweets. The cake (¥1000) is said to be popular." (output conversation statement 4406). In this example, the conversational AI extracts "DDD," a dining-related facility located ahead of user U1's direction of travel, as a candidate and suggests recommended dining locations based on information about the facility (either map data or web search information), user U1's attributes, and preferences.

[0482] In addition, conversational AI and apps can generate URLs and other links based on conversational phrases. Users can also obtain detailed information related to a specific word in a conversational phrase by clicking on it or by inputting it via voice. For example, a link to "DDD" or "cake" can provide detailed information about the shop or food in question.

[0483] User U1 responds to output session statement 4406, for example, agreeing and uttering something like "Looks good. Let's go check it out." (input session statement 4407).

[0484] As in the specific example above, topics can be provided based on the surrounding circumstances of user U1, and different topics can be changed based on user U1's response, thus enabling a conversation with AI.

[0485] [Example of a conversation screen]

[0486] Figure 45 As a supplement, an example of a GUI for a conversation with AI is shown in the front screen of the smartphone 1A. Figure 45 An example is shown where the text of the conversation (including the text of the voice recognition results) is displayed. Figure 45 In the middle, on the touch panel screen 4500, there is an internal camera (camera C2) that serves as the interface for "AI chat." There is a display bar 4501 showing the AI ​​character's response dialogue and a display bar 4502 showing the user's input dialogue. Additionally, at the bottom, a map 4503 displays the map when using a map app (navigation function). Furthermore, map 4503 can also be configured as other windows.

[0487] [Methods for expressing facial expressions / emotions]

[0488] As a method to reflect facial expressions / emotions in AI conversations, any of the following can also be used.

[0489] (1) AI can also change the topic of the conversation based on the state of facial expressions / emotions.

[0490] (2) AI can also change the way of expressing things in the conversation without changing the topic of the conversation, based on the state of facial expressions / emotions.

[0491] As described above, according to Embodiment 2, even if user U1 does not perform a shooting operation, a conversation with the AI ​​reflecting the surrounding situation of user U1 and device 1 can be achieved. Furthermore, even if the user does not input any conversational phrases, conversational phrases can be automatically generated on the AI ​​side based on the input of camera images.

[0492] [Modification of Implementation Method 2]

[0493] As a variation of implementation method 2, the following method can also be implemented.

[0494] When the input from both the spoken dialogue and the camera image is roughly simultaneous, this is not limited to conversational AI (e.g. Figure 36A The processing of a single response / input / output (355) is performed in a 1:1 ratio for image recognition AI (e.g., Figure 36A The 356) response / input / output processing can be performed, or it can be done at a preset ratio. For example, in the process of... Figure 36A If the processing load of image recognition AI 356 in identifying the surrounding situation from an image is higher than the processing load of generating a session through conversation AI 355, the ratio of conversation AI processing to image recognition AI processing can be set, for example, to 2:1 to suppress the overall load. Conversely, when using... Figure 36A If the processing load of the image recognition AI 356 in recognizing the surrounding situation from the image is lower than the processing load of the session generated by the session AI 355, the ratio of the session AI processing to the image recognition AI processing can be set to, for example, 1:2 to suppress the overall load.

[0495] Regarding the surrounding situation identified from camera images, there are several patterns. For example, there are cases where stationary surrounding targets are detected in the space of a facility, etc., and cases where objects that require attention for pedestrians (moving objects such as cars, bicycles, other people, etc.) are detected, as described in Implementation 1. Priorities / orders can also be set among these surrounding targets. Objects requiring attention are given a relatively high priority. When the conversational AI detects an object requiring attention as a surrounding target in a conversation with the user, it inserts conversational statements indicating that the object should be noted, prioritizing these statements over typical conversational statements (such as topics about the facility).

[0496] Additionally, categories / categories can be set for surrounding targets on the map, and priorities / orders can be set among these categories / categories based on the user's interest / concern. Similarly, the output of conversational statements can be controlled. Examples of categories / categories include history, nature, dining, and shopping. The user's level of interest / concern can also be customized.

[0497] When using both the voice reading function of Implementation 1 (i.e., the function of reading aloud surrounding objects and situations) and the AI ​​conversation function of Implementation 2 (i.e., the function of reflecting the surrounding situation as a topic in the conversation), priorities and processing ratios can be set among these functions. For example, a mode that prioritizes voice reading and a mode that prioritizes AI conversation can be prepared. For example, in the mode that prioritizes voice reading, voice reading automatically continues when there is no conversational input from the user. When there is conversational input from the user, the conversation with the AI ​​is automatically switched, and voice reading is suppressed.

[0498] Even in Implementation 2, similar to Implementation 1, priority / order can be set for the multiple cameras of Device 1. That is, content such as surrounding targets identified from the images of high-priority cameras is reflected in the conversational AI's conversation with priority.

[0499] Even in Implementation 2, just like in Implementation 1, control corresponding to the state of whether device 1 and server 2 can communicate can be applied.

[0500] As a variation, images from only one camera located on device 1 can also be used. For example, only camera C2 (internal camera) of smartphone 1A can be used. The user's face and the surrounding environment behind him (surrounding objects) are captured in the image taken by the internal camera. Server 2 (conversation AI) identifies at least one of the facial expressions, gestures, and surrounding objects from the camera image. The conversation AI reflects the identified content into a response conversational statement.

[0501] As a variation, in such Figure 36B In this structure, device 1 identifies surrounding targets from camera images using image recognition AI 357, obtaining text representing these targets. Furthermore, the session client 353 can also append textual prompts representing the surrounding targets as supplementary prompts to the input session statements. Request data sent from device 1 to server 2 includes, for example,... Figure 38As shown in the modified example b4' at the bottom, in addition to the input session statement prompt 3801, there is also an auxiliary prompt 3803. The session client can also create a prompt that combines the input session statement prompt 3801 and the auxiliary prompt 3803 into one. The session AI on the server side generates a response session statement based on the request b4' which includes such a prompt.

[0502] As a variation, the conversational AI can also, for the input conversational statement, generate, as candidates, an output conversational statement (first conversational statement) that does not reflect surrounding objects identified from the camera image as topics, and an output conversational statement (second conversational statement) that reflects surrounding objects identified from the camera image as topics. The conversational AI selects and outputs the output conversational statement from the candidates based on a predetermined judgment / evaluation. The conversation flow changes according to the selection. Facial expressions can also be used as the predetermined judgment / evaluation. For example, if the expression is happy / pleasant, the conversation on topics based on surrounding objects can be expanded; if the expression is angry / sad, the conversation on topics based on surrounding objects can be terminated, or the topic can be switched to another topic.

[0503] The embodiments of this disclosure have been described in detail above, but are not limited to the above embodiments. Various modifications can be made without departing from the spirit of the matter. In addition to the essential constituent elements, each embodiment can have constituent elements added, deleted, or replaced. Unless otherwise specified, each constituent element can be singular or multiple. It is also possible to combine various embodiments and variations.

[0504] (Symbol Explanation)

[0505] 1: Equipment (image reading device); 2: Server device; 3: Cloud computing system; 9: Communication network; 20: AI; U1: User.

Claims

1. An image-to-speech system comprising a device carried or worn by a user, the device reading aloud an object from an image captured by a camera using sound, wherein, The device automatically and repeatedly captures images from the camera's footage at predetermined time intervals. The captured image is analyzed to obtain information including text representing objects within the image. Based on the information obtained, and through predetermined judgment, the object and text to be read aloud are determined. The device automatically and repeatedly reads aloud the text representing the determined object at predetermined time intervals.

2. The image reading system according to claim 1, wherein, The image capture, which is repeated at predetermined time intervals, is performed continuously at predetermined first time intervals even without user operation. The predetermined timed repeated audio reading is performed continuously at a predetermined second time interval, even without user operation.

3. The image reading system according to claim 1, wherein, On the timeline, the changes of the object between multiple captured images are judged, and the object that becomes the object to be read aloud is determined in a way that prioritizes the object with relatively large changes.

4. The image reading system according to claim 1, wherein, Based on the analysis of the image, the relative distance or relative speed between the device and the object in the image is determined, and the object that becomes the object to be read aloud is determined in a way that prioritizes objects whose relative distance is within a threshold or whose relative speed is above a threshold.

5. The image reading system according to claim 1, wherein, Based on the analysis of the image, considering the direction of movement of the object, the possibility of the user coming into contact with the object is determined. To avoid this contact, a direction of movement towards an empty space with a low probability of contact with the object is determined, and the user is guided to move in the direction of movement towards the empty space.

6. The image reading system according to claim 4, wherein, The smaller the relative distance or the greater the relative speed, the greater the perceived proximity of the object to the user. Regarding objects that are highly approximated, the volume of the voice reading may be increased, the timbre may be changed, or a warning sound may be added, depending on the degree of approximation.

7. The image reading system according to claim 1, wherein, As the object to be read aloud, a specific object is pre-defined. The object that will be read aloud is determined in a way that prioritizes the specific object. The specific objects include pedestrian crossings, traffic lights, and road signs.

8. The image reading system according to claim 7, wherein, Based on the analysis of the image, the signal state of the traffic light is determined as the specific object. If the signal is determined to have changed, the device will read aloud the sound indicating the change in the signal.

9. The image reading system according to claim 1, wherein, The control is as follows: when the sound of the object is repeatedly read aloud at a predetermined time, if there is no significant change in the same object on the time axis, the sound of the same object will not be read aloud.

10. The image reading system according to claim 1, wherein, The control is as follows: when the object is read aloud repeatedly at a predetermined time, if there is no significant change in the same object on the timeline, the same object is read aloud to indicate that there is no change.

11. The image reading system according to claim 1, wherein, When the user inputs a reading instruction to the device, the following actions are performed independently of the automatic image capture and audio reading at predetermined time intervals: The device captures images from the camera's footage. The captured image is analyzed to obtain information including text representing objects within the image. Based on the information obtained, determine the object and text to be read aloud. The device reads aloud the text representing the determined object. and, During the priority execution, even if the object does not change significantly, the text representing the object must be read aloud.

12. The image reading system according to claim 1, wherein, Control is as follows: During the automatic voice reading at a predetermined time, or during the voice reading when the user inputs a reading instruction to the device, In the case of a second or subsequent reading of the same object, the text is read aloud using a different representation of the object compared to the first reading.

13. The image reading system according to claim 1, wherein, The camera in the device has a 360-degree shooting capability. Images are obtained from the camera footage that captures 360-degree images, taking the user's orientation as a reference, including images of the front, back, left, and right sides.

14. The image reading system according to claim 1, wherein, The device has multiple cameras with different shooting directions. The device automatically and repeatedly captures images from the footage of the plurality of cameras at predetermined time intervals.

15. The image reading system according to claim 14, wherein, The plurality of cameras include a front camera that captures images of the front based on the user's orientation and a rear camera that captures images of the rear.

16. The image reading system according to claim 14, wherein, The plurality of cameras include a front camera that captures images of the front based on the user's orientation, a rear camera that captures images of the rear, a left camera that captures images of the left, and a right camera that captures images of the right.

17. The image reading system according to claim 1, wherein, The device has multiple cameras with different shooting directions. For each of the plurality of cameras, a predetermined timing is set for capturing the image and a predetermined timing is set for performing audio reading based on the image from that camera.

18. The image reading system according to claim 17, wherein, Among the plurality of cameras, for the first camera, a first period is set as a predetermined timing for the capture and sound reading; for the second camera, a second period longer than the first period is set as a predetermined timing for the capture and sound reading. If the object or a significant change in the object is detected in the image from the second camera, the second period is set for the first camera and the first period is set for the second camera.

19. The image reading system according to claim 15, wherein, For the front camera, a first cycle is set as a predetermined timing for the capture and audio reading. For the rear camera, a second cycle, the same as the first cycle, is set as a predetermined timing for the capture and audio reading. The system is controlled to process the capture and audio reading of the front camera and the capture and audio reading of the rear camera alternately at different timings.

20. The image reading system according to claim 1, wherein, The device has multiple cameras with different shooting directions. The device automatically and repeatedly captures images from the footage of the multiple cameras at predetermined time intervals. On the timeline, changes in the object across multiple captured images are assessed, and the object that becomes the subject of the reading is determined by prioritizing the object with the most significant changes. For each of the plurality of cameras, a threshold is set to determine the magnitude of the change.

21. The image reading system according to claim 1, wherein, The device has multiple cameras with different shooting directions. The device automatically and repeatedly captures images from the footage of the multiple cameras at predetermined time intervals. For each of the plurality of cameras, a different timbre is used when reading aloud the audio based on the image from that camera.

22. The image reading system according to claim 1, wherein, The device has a three-dimensional sound output device for performing the sound reading. When the sound of the object is read aloud, three-dimensional sound is output from the three-dimensional sound output device in a manner that sounds as if the sound is being generated to the user from the position of the object in three-dimensional space.

23. The image reading system according to claim 1, wherein, The device is at least one of a smartphone, smartwatch, tablet, smart glasses, or head-up display.

24. The image reading system according to claim 1, wherein, A server device having a communication connection with the aforementioned device. The server device performs processing to parse the captured image and obtain information including text representing objects within the image.

25. The image reading system according to claim 1, wherein, When the device can communicate with the server device, it enables the server device to perform the parsing; when it cannot communicate with the server device, it performs a simplified parsing within the device as the parsing. The simplified parsing refers to parsing with a longer cycle than the parsing in the server device, or parsing images from fewer cameras than the number of cameras used in the parsing in the server device.

26. An image reading device, which is the device in an image reading system equipped with a device carried or worn by a user, wherein the device reads aloud an object from an image captured by a camera using sound, wherein... The device automatically and repeatedly captures images from the camera's footage at predetermined time intervals. The image reading system parses the captured image to obtain information including text representing objects within the image. The image reading system, based on the acquired information and through predetermined judgments, determines the object and text to be read aloud. The device automatically and repeatedly reads aloud the text representing the determined object at predetermined time intervals.

27. An image reading method in an image reading system, the image reading system comprising a device carried or worn by a user, the device reading aloud an object from an image captured by a camera using sound, wherein, The image reading method comprises the following steps: The device automatically and repeatedly captures images from the camera's footage at predetermined time intervals; The image reading system parses the captured image to obtain information including text representing objects within the image; The image reading system, based on the acquired information and through predetermined judgments, determines the object and text to be read aloud; and The device automatically and repeatedly reads aloud the text representing the object of the decision at predetermined time intervals.

28. A response output system comprising a device carried or worn by a user, for responding to the user, wherein, The device has multiple cameras with different shooting ranges, automatically and repeatedly captures multiple images from the multiple cameras at set time intervals, and generates input dialogue statements based on user input, then inputs the input dialogue statements and the multiple images into the interface. The interface outputs the generated output session statement based on the input session statement and the identification of the user's surroundings based on image recognition processing of the multiple images.

29. The response output system according to claim 28, wherein, The interface inserts the generated output session statement into the flow of the session, the output session statement taking the surrounding targets detected from the surrounding conditions of the images captured by at least one camera in the plurality of images as topics.

30. The response output system according to claim 28, wherein, The interface inserts the generated output session statement into the flow of the session, and the output session statement uses the facial expressions of the user identified from images captured by at least one camera in the plurality of images as topics.

31. The response output system according to claim 28, wherein, The interface outputs the generated output session statement, which uses surrounding targets detected from the surrounding conditions of images captured by at least one camera in the plurality of images as topics. Furthermore, the interface controls whether to output a conversational statement that uses the surrounding target as a topic, or whether to change it to an output conversational statement that uses other surrounding targets as topics, based on the facial expressions of the user identified from the images captured by at least one camera among the plurality of images, and inserts the output conversational statement into the flow of the conversation.

Images