Data processing device, wearable terminal, data processing method, and program

Abstract

To provide a data processing device capable of providing a user having color blindness with appropriate visual support, a wearable terminal, a data processing method, and a program.SOLUTION: A data processing device includes an input unit which accepts an image including at least a part of a visual field of a user, a processing unit which performs specific processing using a data generation model for generating data according to input contents, and an output unit which outputs color blindness support information, which is information supporting visual perception associated with the color blindness of the user according to the processing results by the processing unit. The processing unit acquires color blindness support information as the processing results using an output of a data generation model when input contents including an instruction sentence instructing the generation of a correction color for correcting the color of an object included in the image are input.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The technology disclosed herein relates to a data processing device, a wearable terminal, a data processing method, and a program. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

[0003] Patent Document 2 describes an image display device that displays an image on the windshield of a vehicle, characterized in that it comprises: image capturing means for capturing an image in front of the vehicle; recognition target detection means for detecting a recognition target that the driver should recognize in front of the vehicle; background color detection means for detecting a background color of the recognition target as seen by the driver based on the recognition target detected by the recognition target detection means and the image captured by the image capturing means; and color adjustment image display means for superimposing and displaying a color adjustment image on the windshield that adjusts the color of the recognition target as seen by the driver based on the driver's color vision characteristic information and the background color detected by the background color detection means. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-180282 A [Patent Document 2] JP 2012-121527 A Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 describes a technology related to a persona chatbot that responds to user utterances. Patent Document 1 only responds to user utterances, and does not take into consideration providing support (for example, visual support) to the user.

[0006] Furthermore, Patent Document 2 describes a technology that provides visual support to a user by superimposing a color-adjusted image on the windshield that adjusts the color of a recognized object as seen by the driver. However, while Patent Document 2 discloses displaying an image in which the color of a recognized object is adjusted based on the background color and the driver's color vision characteristic information, it does not disclose how the color is adjusted specifically. In other words, there are cases in which an image with an appropriate color adjustment is not displayed to the user.

[0007] As such, the conventional technology has room for improvement in terms of providing appropriate visual support to users with color vision deficiency. [Means for solving the problem]

[0008] A first aspect of the technology disclosed herein includes an input unit that accepts an image including at least a portion of a user's visual field, a processing unit that performs specific processing using a data generation model that generates data according to the input content, and an output unit that outputs color vision deficiency support information, which is information that supports the visual cognition of a user with color vision deficiency, according to a processing result by the processing unit, wherein the processing unit is a data processing device that obtains the color vision deficiency support information as a processing result using the output of the data generation model when input content including an instruction statement that instructs the generation of a correction color to correct the color of an object included in the image is input.

[0009] A second aspect of the technology disclosed herein is a data processing method that includes accepting an image including at least a portion of a user's visual field, performing a specific process using a data generation model that generates data according to the input content, and outputting color vision deficiency support information, which is information that supports the visual cognition of a user with color vision deficiency, according to the processing result of the specific process, wherein the color vision deficiency support information is obtained as a processing result using the output of the data generation model when input content including an instruction statement that instructs the generation of a correction color to correct the color of an object included in the image is input in the specific process.

[0010] A third aspect of the technology disclosed herein is a program that causes a computer to execute a process that includes receiving an image including at least a portion of a user's visual field, performing a specific process using a data generation model that generates data according to the input content, and outputting color vision deficiency support information, which is information that supports the visual cognition of a user with color vision deficiency, according to a result of the specific process, wherein the color vision deficiency support information is obtained as a processing result using the output of the data generation model when input content including an instruction statement that instructs the generation of a correction color to correct the color of an object included in the image is input in the specific process. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Diagram 2] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a smart device. [Diagram 3] 2 shows a schematic functional configuration of a specific processing unit of the data processing device. [Figure 4] 10 is a flowchart illustrating an example of an operational flow of a specific process by a data processing device. [Diagram 5] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device. [Figure 6] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device. [Figure 7] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device. [Figure 8] 10 is a flowchart showing an example of the flow of a specification process by the data processing device. [Figure 9] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device. [Figure 10] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0013] First, the terms used in the following description will be explained.

[0014] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0015] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0016] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0017] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0019] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

[0020] As shown in FIG. 1, the data processing system 10 includes a data processing device 12 and smart glasses 14. An example of the data processing device 12 is a server. The smart glasses 14 are glasses-type terminals that can be worn by a user 20. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure. Also, the smart glasses 14 are an example of a "wearable terminal" and a "glasses-type terminal" according to the technology of the present disclosure.

[0021] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0022] The smart glasses 14 include a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0023] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The processor 46 transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12 via the communication I / F 44. In the data processing device 12, the processor 28 acquires the data indicating the user input via the communication I / F 44.

[0024] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0025] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0026] FIG. 2 shows an example of main functions of the data processing device 12 and the smart glasses 14.

[0027] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0028] The storage 32 stores a data generation model 58. The data generation model 58 is used by the specific processing unit 290.

[0029] In the smart glasses 14, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0030] Next, the process of the specific processing unit 290 when the data processing device 12 performs specific processing for correcting the color of the captured image obtained by the smart glasses 14 will be described.

[0031] As shown in FIG. 3, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

[0032] The input unit 292 acquires a user input accepted by the smart glasses 14. Specifically, the input unit 292 acquires at least one of text, voice, and image data of the user accepted by the smart glasses 14.

[0033] The processing unit 294 performs a specific process using the data generation model 58. Specifically, character, voice and / or image data input by the user is input to the data generation model 58, and a generation result is obtained.

[0034] The output unit 296 transmits the result of the specific processing to the smart glasses 14. In the smart glasses 14, the control unit 46A causes the output device 40 to output the result of the specific processing.

[0035] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction sentence is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 infers the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization. The data generation model 58 is an example of a "data generation model" related to the technology of the present disclosure.

[0036] Next, the operation of the data processing system 10 will be described.

[0037] An example of the flow of the identification process will be described with reference to FIG.

[0038] In step S300, the processing unit 294 determines whether a predetermined trigger condition is met.

[0039] If the trigger condition is met in step S300 (step S300; Yes), the data processing system 10 proceeds to step S301. On the other hand, if the trigger condition is not met in step S300 (step S300; No), the data processing system 10 ends the identification process.

[0040] In step S301, the processing unit 294 generates a prompt by adding an instruction for obtaining a result of a specific process to the text representing the input.

[0041] In step S303, the processing unit 294 inputs the generated prompt to the data generation model 58, and obtains the result of the specific process based on the output of the data generation model 58.

[0042] In step S304, the output unit 296 outputs the result of the identification process to the user terminal, and the identification process ends.

[0043] Incidentally, it is estimated that there are currently about 3 million people in Japan who have color vision deficiency (i.e., color vision characteristics different from general color vision) and experience inconvenience in their daily lives. In the case of color vision deficiency, compared to people with general color vision, there is an abnormality in the eye tissue (i.e., the cones) that senses any of the colors green, red, or blue, so that the person may see colors differently from people with general color vision. Therefore, people with color vision deficiency experience the following inconveniences due to their color vision deficiency. For example, they may experience inconveniences such as difficulty in understanding the color of traffic lights, being unable to distinguish the colors of ingredients or food, failing to select or cook ingredients, or difficulty in understanding the colors of clothes, which means they cannot purchase their favorite clothes.

[0044] Color vision deficiency can also limit the range of career options available to people with color vision deficiency. For example, people with color vision deficiency may find it difficult to work as a designer, who requires a high level of color sensitivity, or as a train driver, who must accurately judge the colors of traffic lights.

[0045] In this embodiment, visual support is provided to people with such color vision deficiency using the data processing system 10. Specifically, a specific process is executed in the data processing system 10, thereby providing information for supporting the visual cognition of the user with color vision deficiency.

[0046] The specific processing of this embodiment has the following functions to achieve the following objectives: That is, in order to address the issue that a user 20 who has or is prone to color vision deficiency has difficulty in accurately recognizing the color of an object compared to general color recognition, in the identification process, first, an object detection process is executed on a captured image (e.g., a still image or a video) acquired by the camera 42 mounted on the smart glasses 14 to detect an image area indicating a target object contained in the image. Here, in the object detection process, an object detection and object recognition are performed in the image using a trained model for object detection. For example, a YOLO (You Only Look Once) object detection algorithm is used in the trained model for object detection. Next, in the identification process, a process is executed to generate a prompt including an instruction sentence for the data generation model 58 based on information indicating an object in the image obtained in the object detection process and color information of the detected object. In this process, the prompt is generated based on information indicating the degree and state of the color vision deficiency of the user 20 (for example, which colors are difficult to see). Then, in the identification process, the generated prompt is input to the data generation model 58, and a process of obtaining an answer is executed. The data generation model 58 generates an answer as to how to correct the color of the object in the captured image according to the prompt in accordance with the user's color vision deficiency. Then, the correction color information output from the data generation model 58 is acquired. The correction color information is information indicating how to correct the color of an image region showing an object in the captured image. Furthermore, in the identification process, a process is executed to display an image of the target object with the color corrected on display 40A. That is, an image with the color corrected is displayed at a position corresponding to the target object in the scene currently being viewed by the user.

[0047] Next, the specific processing of this embodiment will be specifically described with reference to FIGS.

[0048] As an example, as shown in FIG. 5, first, an image including at least a part of the visual field of the user 20 is obtained by the camera 42 mounted on the smart glasses 14. Here, an example is shown in which an image 42A including a range equivalent to the visual field of the user 20 (hereinafter also simply referred to as "visual field range image 42A") is obtained. In this case, the angle of view of the camera 42 is set in advance according to a general visual field angle of a human. In addition, "corresponding to the visual field of the user" includes a case where the visual field is exactly the same as that of the user 20, and also includes a case where the visual field of the user 20 is included and has a wider range than that. In the example shown in FIG. 5, the visual field range image 42A corresponds to the visual field of the user 20 when the user 20 is looking at a traffic light.

[0049] Note that, here, a case where the visual field range image 42A is obtained by the camera 42 is taken as an example, but this is merely one example. The camera 42 may acquire an image showing a part of the visual field of the user 20. For example, the camera 42 may acquire an image of a partial range of the visual field of the user 20 toward which the gaze of the user 20 is directed (for example, an image of a part where the user 20 is gazing).

[0050] In the present embodiment, the visual field range image 42A includes a still image and a moving image. When the visual field range image 42A is a still image, the camera 42 captures the image at a predetermined timing (for example, every 0.1 seconds). The still image may be an image of multiple frames (for example, three frames).

[0051] In the processor 46 of the smart glasses 14, the control unit 46A acquires the field of view range image 42A from the camera 42. Furthermore, the control unit 46A transmits the acquired field of view range image 42A to the data processing device 12 via the communication I / F 44 (see FIG. 1).

[0052] In the processor 28 of the data processing device 12, the input unit 292 accepts the field of view range image 42A transmitted from the smart glasses 14. Then, the input unit 292 outputs the acquired field of view range image 42A to the processing unit 294. The processing unit 294 performs a specification process using the data generation model 58. First, the processing unit 294 executes an object detection process on the field of view range image 42A using the trained model 62 for object detection.

[0053] Specifically, the storage 32 stores a trained model 62 for object detection. The trained model 62 for object detection is obtained by optimizing a neural network by performing machine learning using training data on the neural network. The training data is a plurality of data in which example data and correct answer data are associated with each other (i.e., data in which image data and annotations of objects in the image are associated with each other). The example data is image data, and the correct answer data is annotations of detected objects corresponding to the example data. The trained model 62 for object detection is an example of a "trained model for object detection" according to the technology of the present disclosure.

[0054] Here, the trained model for object detection 62 may be, for example, a trained model that performs object detection and class classification in an image using the YOLO method. The trained model for the YOLO method is capable of high-speed object detection (for example, about several tens of ms per image) compared to other trained models for object detection.

[0055] The processing unit 294 acquires the trained model for object detection 62 from the storage 32. The processing unit 294 inputs the acquired field of view range image 42A to the trained model for object detection 62. When the field of view range image 42A is input, the trained model for object detection 62 outputs an object detection result 64 which is a result of detecting an object in the field of view range image 42A. The processing unit 294 acquires the object detection result 64.

[0056] The object detection result 64 includes a classification result of the detected object, the color of the detected object, and the size and position of the detected object in the image. In the example shown in FIG. 5, the object detection result 64 indicates that a "traffic light" is detected in the field of view range image 42A and that the color of the traffic light is "red." Here, in the object detection result 64, the color of the detected object may be indicated by RGB values. Also, in the object detection result 64, the position and size of the traffic light and the position and size of the lit part are indicated by a bounding box 64A in the field of view range image 42A.

[0057] Although only a traffic light is detected in the visual field range image 42A here, this is merely an example. Of course, a plurality of objects may be detected in the visual field range image 42A.

[0058] As an example, as shown in FIG. 6, the processing unit 294 executes the corrected color generation process after the object detection process. The processing unit 294 acquires the object detection result 64 from the trained model 62 for object detection. The processing unit 294 also acquires the user color vision information 66 from the storage 32. The user color vision information 66 is information capable of identifying the color vision characteristics of the user 20. Here, the color vision characteristics of the user 20 refer to the characteristics of the color vision of the user 20 compared to the general color vision. For example, when the user 20 has difficulty seeing red, the color vision characteristics of the user 20 may be a characteristic that "only 60% of red can be seen compared to the general color vision." Here, "only 60% of red can be seen" means that only 60% of the cones that sense red exist compared to a person with general color vision.

[0059] The user color vision information 66 is generated in advance based on, for example, the results of a doctor's diagnosis of the color vision of the user 20 (for example, the results of a diagnosis of which colors are difficult to see and to what extent), and is stored in the storage 32.

[0060] Here, the user color vision information 66 is stored in the storage 32, but this is merely one example. The user color vision information 66 may be stored in an external device (e.g., an external server) of the data processing device 12 and transmitted from the external device, or may be transmitted from the smart glasses 14 to the data processing device 12.

[0061] Then, the processing unit 294 uses the object detection result 64 and the user color vision information 66 to generate instructions (i.e., a prompt) for obtaining data for correction color generation processing. The prompt includes the detected object indicated by the object detection result 64, the color of the detected object, and the color vision characteristics of the user 20 indicated by the user color vision information 66. The prompt includes an instruction sentence for generating a correction color based on the color of the detected object. The prompt also includes an instruction sentence for causing the data generation model 58 to generate a correction color based on the color vision characteristics of the user 20. Here, the correction color is a color for correcting the color of the detected object.

[0062] For example, the following prompt might be generated: "A user with color vision deficiency is looking at a traffic light. The traffic light has a red light (RGB values, R:XX, G:YY, B:ZZ) on. The user has a color vision deficiency in which they can only see 60% of the red color compared to normal color vision. Generate RGB values ​​of a color that is corrected to be easier for the user to see, based on the current RGB values."

[0063] Furthermore, when correcting the color of a detected object, it is possible to align the appearance of a color other than the color that is difficult for the user 20 to see with the color that is difficult for the user 20 to see. This will result in the appearance of the three colors red, green, and blue being in the same balance, and the appearance of colors by the user 20 can be made closer to that of general color vision.

[0064] For example, in the case of user 20 who can only sense red at 60%, when seeing a color that generally looks yellow, the degree to which the user senses red is weak, and the color is perceived as a greenish yellow (i.e., yellow-green). In this case, if the degree to which the user senses green is suppressed to 60%, the degree to which the user senses red and green will be balanced, and the way the user 20 sees yellow can be brought closer to the way the user generally sees yellow.

[0065] It is possible to reflect this idea in the prompt. In this case, the prompt might look something like this: "A user with color vision deficiency is looking at a traffic light. The traffic light has a red light (RGB values, R:XX, G:YY, B:ZZ) on. The user has a color vision deficiency in which they can only see 60% of the red color compared to normal color vision. Generate RGB values ​​of colors that are corrected so that the user can see them easily. When correcting the colors, correct the G and B values ​​of the RGB values ​​so that they are closer to how the user sees the R, which is difficult to see."

[0066] The processing unit 294 inputs the generated prompt to the data generation model 58. The processing unit 294 acquires information indicating the correction color from the data generation model 58. The information indicating the correction color is, for example, the RGB value of the correction color.

[0067] The processing unit 294 generates the corrected image 68 using the output of the data generation model 58 when the above-mentioned prompt is input. The corrected image 68 is an example of the "color vision deficiency support information" and the "corrected image" according to the technology of the present disclosure. The corrected image 68 is a partial image showing a detected object whose color has been corrected with a correction color. Specifically, the processing unit 294 uses the position and size of the detected object in the object detection result 64 to identify an image area in which the color is to be corrected. Then, the processing unit 294 sets the pixel value of each pixel in the partial image corresponding to the identified image area to the RGB value of the correction color acquired from the data generation model 58. In this way, the processing unit 294 generates the corrected image 68. In the example shown in FIG. 6, an image showing a red light part of a traffic light and corrected with a correction color is shown as the corrected image 68. The processing unit 294 outputs the generated corrected image 68 to the output unit 296. In this way, information for supporting visual cognition with color vision deficiency is output from the output unit 296.

[0068] As an example, as shown in FIG. 7, in the data processing device 12, the output unit 296 transmits the corrected image 68 to the smart glasses 14 via the communication I / F 44 (see FIG. 1). In the processor 46 of the smart glasses 14, the control unit 46A acquires the corrected image 68. Then, the control unit 46A causes the display 40A to display the acquired corrected image 68. Specifically, the control unit 46A controls a GUI (Graphical User Interface) for displaying the corrected image 68, thereby causing the display 40A to display the corrected image 68. Here, the display 40A is provided at a position corresponding to the lens of the smart glasses 14, and is a display device capable of displaying an image to the user 20. In the display 40A, light from the outside can be transmitted through a portion other than the area where the image is displayed. The display 40A is an example of a "display device" according to the technology of the present disclosure.

[0069] In the corrected image 68, the position of the portion where the color is corrected is set based on the visual field range image 42A. Moreover, the visual field range image 42A corresponds to the visual field seen by the user 20. Therefore, the portion where the color is corrected in the corrected image 68 corresponds to the position of the portion where the color should be corrected in the scene actually seen by the user 20. That is, in the scene seen by the user 20 through the smart glasses 14, the corrected image 68 is superimposed on the display 40A, so that the color that is difficult for the user 20 to see is corrected in the visual field of the user 20.

[0070] Next, the visual support processing in the data processing device 12 according to the present embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the visual support processing. The flow of the processing shown in Fig. 8 is an example of a "data processing method" according to the technology of the present disclosure.

[0071] As an example, in the visual support processing shown in FIG. 8, first, in step ST10, the processing unit 294 determines whether or not a predetermined trigger condition is satisfied. Here, an example of the predetermined trigger condition is that an instruction to start the visual support processing is accepted. If the trigger condition is satisfied in step ST10, the determination is affirmative, and the visual support processing proceeds to step ST12. If the trigger condition is not satisfied, the determination is negative, and the visual support processing repeats the processing of step ST10. After the processing of step ST10 is executed, the visual support processing proceeds to step ST12.

[0072] In step ST12, the input unit 292 acquires the visual field range image 42A from the smart glasses 14. After the processing of step ST12 is executed, the visual support processing proceeds to step ST14.

[0073] In step ST14, the processing unit 294 inputs the visual field range image 42A acquired in step ST12 to the trained model for object detection 62. After the processing of step ST14 is executed, the visual support processing proceeds to step ST16.

[0074] In step ST16, the processing unit 294 acquires the object detection result 64 output from the trained model for object detection 62. After the processing of step ST16 is executed, the visual support processing proceeds to step ST18.

[0075] In step ST18, the processing unit 294 generates a prompt including an instruction to generate a corrected color of the object in the visual field range image 42A detected in step ST16. After the processing of step ST18 is executed, the visual support processing proceeds to step ST20.

[0076] In step ST20, the processing unit 294 inputs the prompt generated in step ST18 to the data generation model 58. After the processing in step ST20 is executed, the visual support processing proceeds to step ST22.

[0077] In step ST22, the processing unit 294 acquires the corrected color output in step ST20 from the data generation model 58. After the processing in step ST22 is executed, the visual support processing proceeds to step ST24.

[0078] In step ST24, the processing unit 294 uses the corrected color acquired in step ST22 to generate the corrected image 68. After the processing of step ST24 is executed, the visual support processing proceeds to step ST26.

[0079] In step ST26, the output unit 296 outputs the corrected image 68 generated in step ST24 to the smart glasses 14. After the processing of step ST26 is executed, the visual support processing proceeds to step ST28.

[0080] In step ST28, the processing unit 294 determines whether or not a condition for terminating the visual support processing (hereinafter referred to as the "termination condition") has been satisfied. One example of the termination condition is that an instruction to terminate the visual support processing has been accepted. If the termination condition has not been satisfied in step ST28, the determination is negative, and the visual support processing proceeds to step ST10. If the termination condition has been satisfied in step ST28, the determination is positive, and the visual support processing terminates.

[0081] As described above, in the data processing system 10 according to this embodiment, in the processor 28 of the data processing device 12, the input unit 292 acquires the visual field range image 42A from the smart glasses 14. Then, the processing unit 294 acquires a correction color, which is a color for correcting the color of an object (e.g., a traffic light) included in the visual field range image 42A, using the data generation model 58. Here, a prompt including an instruction for generating the correction color is input to the data generation model 58. Furthermore, the processing unit 294 generates a correction image 68 corrected with the correction color based on the output result from the data generation model 58. The output unit 296 outputs the correction image 68 to the smart glasses 14. This makes it possible to provide the user 20 with color vision deficiency with a way of seeing an object close to that of general color vision, thereby realizing appropriate visual support for the user 20.

[0082] For example, it is possible to display an image in which the color of the recognized object is adjusted using a conventionally known image processing technique for a user with color vision deficiency. For example, it is possible to display an image in which the color of the recognized object is adjusted based on the background color and the driver's color vision characteristic information. However, in this case, it is necessary to set in advance how to adjust the color, and once the color adjustment method is set, it is troublesome to change the setting again. In this configuration, since the correction color is generated by inputting a prompt to the data generation model 58, it is possible to generate correction colors according to various situations by changing the contents of the prompt (e.g., instruction sentences, etc.). As a result, appropriate visual support can be more effectively provided to users with color vision deficiency.

[0083] Moreover, in the data processing system 10 according to the present embodiment, the processing unit 294 executes an object detection process on the visual field range image 42A and acquires the object detection result 64. Then, the processing unit 294 generates a prompt including information indicating the detected object as a prompt to be input to the data generation model 58. For example, a prompt including a content that a traffic light has been detected in the visual field range image 42A is generated. This makes it possible to accurately identify what kind of object is included in the visual field range image 42A, and further to instruct the data generation model 58 to generate a correction color according to the detected object. As a result, the accuracy of the correction color of the object included in the visual field range image 42A is improved, and appropriate visual support for the user 20 is realized.

[0084] In addition, in the data processing system 10 according to the present embodiment, an object detection process is performed on the visual field range image 42A, and the color of the object detected in the visual field range image 42A is obtained. For example, when a traffic light included in the visual field range image 42A is detected, it is obtained that the light color of the traffic light is red. The prompt input to the data generation model 58 includes the color of the detected object. This improves the accuracy of the corrected color of the object included in the visual field range image 42A compared to the case where the corrected color is generated regardless of the color of the detected object, and therefore, it is possible to provide appropriate visual support to the user 20.

[0085] Furthermore, in the data processing system 10 according to this embodiment, the processing unit 294 generates a prompt including a content to generate a correction color based on the color of the detected object obtained as a result of the object detection process. This allows the data generation model 58 to generate a correction color based on the color of the detected object. As a result, the accuracy of the correction color of the object included in the visual field range image 42A is improved compared to the case where the correction color is generated regardless of the color of the detected object, so that appropriate visual support is provided to the user 20.

[0086] In addition, in the data processing system 10 according to the present embodiment, an object detection process is performed on the visual field range image 42A, and the position and size of the object detected in the visual field range image 42A within the image are obtained. For example, when a traffic light included in the visual field range image 42A is detected, the red light of the traffic light is displayed by a bounding box. The position and size of the detected object in the visual field range image 42A are used when superimposing and displaying the correction image 68 on the scene seen by the user 20. This improves the accuracy of the display position of the correction image 68 compared to when the correction image 68 is displayed regardless of the position and size of the detected object, and therefore, appropriate visual support can be provided to the user 20.

[0087] Furthermore, in the data processing system 10 according to this embodiment, the object detection process is executed using the trained model for object detection 62. The trained model for object detection 62 is a trained model different from the data generation model 58. This realizes faster object detection process using the trained model for object detection 62. Since faster object detection process contributes to faster overall identification process, it is expected to improve the processing speed of the process of providing visual support to the user 20.

[0088] Moreover, in the data processing system 10 according to this embodiment, the processing unit 294 acquires the user color vision information 66 from the storage 32, and generates a prompt based on the user color vision information 66. That is, the prompt includes the color vision characteristics of the user 20. This allows the data generation model 58 to generate a correction color based on the color vision characteristics of the user 20. As a result, compared to a case where a correction color is generated regardless of the color vision characteristics of the user 20, a correction color according to the way the user 20 sees colors is generated, and therefore appropriate visual support for the user 20 is realized.

[0089] Furthermore, in the data processing system 10 according to this embodiment, the processing unit 294 generates a prompt including a content to generate a correction color based on the color vision characteristics of the user 20. This allows the data generation model 58 to generate a correction color based on the color vision characteristics of the user 20. As a result, compared to a case where a correction color is generated regardless of the color vision characteristics of the user 20, a correction color according to the way the user 20 sees colors is generated, so that appropriate visual support can be provided to the user 20.

[0090] Furthermore, in the data processing system 10 according to this embodiment, the output unit 296 outputs a corrected image 68 to the smart glasses 14. The corrected image 68 is superimposed on the view seen by the user 20 via the display 40A in the smart glasses 14. This allows visual support to be provided without displaying unnecessary text information, and allows the view seen by the user 20 to be closer to the view seen by a person with normal color vision, improving the user experience.

[0091] In addition, in the data processing system 10 according to the present embodiment, the correction image 68 is displayed via the display 40A in the smart glasses 14 as a wearable terminal worn by the user 20. This allows the user 20 to receive visual assistance via the terminal he or she is wearing, making it easier for the user 20 to receive visual assistance in various environments.

[0092] Furthermore, in the data processing system 10 according to this embodiment, the corrected image 68 is displayed via the display 40A mounted on the lens of the smart glasses 14. Therefore, the corrected image 68 can be displayed in the field of view of the user 20. This makes it possible to make the scene seen by the user 20 closer to the scene seen by a person with normal color vision, thereby improving the user experience.

[0093] (Variation 1) In the above embodiment, an example has been described in which the processing unit 294 generates a prompt based on the object detection result 64 and the user color vision information 66, but the technology of the present disclosure is not limited to this. In the present first modified example, a prompt is generated taking into account the situation of the user 20.

[0094] As an example, as shown in FIG. 9, the processing unit 294 acquires an object detection result 64 and user color vision information 66. The processing unit 294 also acquires user situation information 70 transmitted from the smart glasses 14. The user situation information 70 is information capable of identifying the situation of the user 20. The situation of the user 20 includes, for example, the environment around the user 20 or the content of the actions performed by the user 20. In the example shown in FIG. 9, the situation of the user 20 is input by voice via a microphone 38B mounted on the smart glasses 14 and transmitted to the data processing device 12 as user situation information 70. Here, information indicating that the user 20 is driving is transmitted as the user situation information 70.

[0095] The processor 294 generates a prompt based on the object detection result 64, the user color vision information 66, and the user context information 70. The instruction sentence in the prompt includes an instruction to the data generation model 58 to generate a correction color based on the context of the user 20 indicated by the user context information 70.

[0096] For example, the following prompt might be generated: "A user with color vision deficiency is looking at a traffic light. The red light (RGB values, R:XX, G:YY, B:ZZ) is on at the traffic light. The user has a color vision deficiency where they can only see 60% of the red light compared to normal color vision. Generate RGB values ​​of colors that are corrected so that the user can easily see them. The user is driving, so generate colors that make it easy to distinguish the color of the traffic light."

[0097] Note that, although an example in which the user 20 is driving has been described here, this is merely one example. For example, the fact that the user 20 is shopping or cooking may be input as the content of the user 20's action. In this case, the prompt may include an instruction to correct the color of the product that the user 20 is picking up to purchase so that it is recognizable as the color that the user 20 prefers. Also, the prompt may include an instruction to correct the color of the ingredients that the user 20 is cooking so that their freshness can be recognized. Also, the current time period (e.g., daytime or nighttime) or the weather (e.g., sunny or rainy) may be input as the situation of the user 20.

[0098] The processing unit 294 generates a corrected image 68 based on the output result of the data generation model 58, and outputs the generated corrected image 68 to the output unit 296. In the smart glasses 14, the control unit 46A causes the corrected image 68 acquired from the output unit 296 to be superimposed and displayed on the display 40A (see FIG. 7).

[0099] As described above, in this first modified example, the processing unit 294 generates a prompt based on the user situation information 70. This allows the data generation model 58 to generate a correction color based on the situation of the user 20. As a result, compared to a case where a correction color is generated regardless of the situation of the user 20, a correction color is generated according to the situation of the user 20, so that appropriate visual support can be provided to the user 20.

[0100] For example, when the user 20 is driving, he / she may be required to pay attention to the color of a traffic light. For this reason, the prompt input to the data generation model 58 includes, for example, an instruction to generate a correction color so that the user 20 can easily distinguish the color of the traffic light when the user 20 is driving. Then, the correction image 68 is generated using the output result from the data generation model 58, so that appropriate visual support can be provided to the user 20.

[0101] Also, for example, it is possible to execute the object detection process using a trained model for object detection specialized for object detection in a specific scene. However, in this first modified example, various objects included in the visual field range image 42A can be detected by not limiting the situation of the user 20 at the stage of the object detection process. Then, the prompt includes an instruction sentence instructing how to generate a correction color for an object detected by the object detection process in consideration of the situation of the user 20. This realizes generation of an appropriate correction color according to the situation of the user 20 while detecting various objects. In addition, it is not necessary to re-tune the trained model for object detection according to the situation of the user 20 or the object to be detected, and the cost required for fine-tuning the trained model can be reduced.

[0102] In the first modified example, the user situation information 70 is input by voice input, but the technology of the present disclosure is not limited to this. For example, the user situation information 70 may be received via the touch panel 38A. Alternatively, the control unit 46A may estimate the situation of the user 20 based on the output results from various sensors (e.g., an acceleration sensor, a temperature and humidity sensor, or an optical sensor) mounted on the smart glasses 14 and / or the camera 42, and generate the user situation information 70.

[0103] (Variation 2) In the above embodiment, an example in which a general-purpose trained model for object detection 62 is used in the object detection process has been described, but the technology of the present disclosure is not limited to this. In the second modification, one trained model is selected from a plurality of trained models for object detection in response to a selection by the user 20, and is used in the object detection process.

[0104] As an example, as shown in FIG. 10, a field of view range image 42A is obtained by a camera 42 mounted on a smart glass 14. The control unit 46A transmits the field of view range image 42A to the data processing device 12. The control unit 46A also receives a selection of an object detection mode by the user 20 via the reception device 38 (see FIG. 1). Then, the control unit 46A transmits mode selection information 72, which is information indicating the selected object detection mode, to the data processing device 12. The object detection mode is a mode that specifies a scene to be subject to object detection. Here, it is assumed that a driving mode is selected as the object detection mode.

[0105] In the processor 28 of the data processing device 12, the input unit 292 acquires the field of view range image 42A and the mode selection information 72 from the smart glasses 14. Then, the input unit 292 outputs the field of view range image 42A and the mode selection information 72 to the processing unit 294. The processing unit 294 first determines a trained model for object detection to be used for the object detection process according to the object detection mode indicated by the mode selection information 72.

[0106] The processing unit 294 determines a trained model according to the object detection mode, for example, by using a trained model selection table (not shown). Here, the trained model selection table is a table in which the object detection mode is input information and information capable of identifying a trained model according to the object detection mode is output information.

[0107] A plurality of trained models for object detection are stored in advance in the storage 32. Machine learning is performed on the plurality of trained models for object detection according to each of a plurality of scenes in which the model is expected to be used. Here, an example is shown in which a first trained model for object detection 62A and a second trained model for object detection 62B are stored in the storage 32.

[0108] The first object detection trained model 62A is, for example, a trained model for object detection specialized for detecting objects while driving a car. The first object detection trained model 62A is obtained by performing machine learning on a neural network using a data set as teacher data in which example data is image data showing an image while driving and correct answer data is an annotation of a detected object corresponding to the example data.

[0109] The second trained model 62B is a trained model for object detection specialized for detecting objects during cooking, for example. The second trained model for object detection 62B is obtained by performing machine learning on a neural network using a data set as training data in which example data is image data showing an image during cooking and correct answer data is an annotation of a detected object corresponding to the example data.

[0110] Then, the processing unit 294 acquires the trained model for object detection from the storage 32. As described above, since the driving mode is selected here, in the example shown in FIG. 10, the trained model for object detection 62A specialized for object detection while driving is acquired from the storage 32. Then, the processing unit 294 executes object detection processing on the field of view range image 42A using the first trained model for object detection 62A. The processing unit 294 acquires the object detection result 64 output from the first trained model for object detection 62A.

[0111] The processing unit 294 executes a corrected color generation process based on the object detection result 64. Then, the processing unit 294 generates a corrected image 68 using the corrected color obtained by the corrected color generation process, and the output unit 296 outputs the corrected image 68 to the smart glasses 14, and the corrected image 68 is superimposed and displayed on the smart glasses 14 via the display 40A (see FIG. 7).

[0112] As described above, in the second modified example, the processing unit 294 selects a trained model for object detection according to the mode selection information 72 from among a plurality of trained models for object detection. For example, when the mode indicated by the mode selection information 72 is a driving mode, the first trained model for object detection 62A specialized for object detection while driving is selected. This enables object detection according to the mode selected by the user 20, and improves the accuracy of object detection (e.g., the accuracy of object identification, the accuracy of object size and position, and / or the accuracy of object color). As a result, the accuracy of the corrected color in the corrected color generation process and / or the accuracy of the correction points in generating the corrected image 68 are also improved.

[0113] For example, consider a case where the user 20 draws a picture or buys clothes. In such a case, while the objects to be detected are limited to a certain extent, it is required to accurately identify various colors. Therefore, by executing the object detection process using a trained model for object detection that is specialized for object detection in each situation according to the object detection mode selected by the user 20, it is possible to perform object detection (for example, detection of the color of an object) with higher accuracy than when a trained model for general-purpose object detection is used. This allows more specific instructions to be included in the generation of correction colors in the instruction statement for the data generation model 58. For example, the RGB values ​​of the color of the detected object can be specified more finely, and the accuracy of the correction color generated by the data generation model 58 can be improved. As a result, the accuracy of the visual support provided to the user 20 is also improved.

[0114] Furthermore, the trained model for object detection according to this modification may be not only a trained model stored in advance, but also a trained model that has been retrained in response to feedback from a user. That is, input from a user regarding the object detection result (e.g., an answer input regarding the success or failure of the detection result, or an annotation by the user regarding the detection result) may be accepted, and retraining may be performed using the input from the user as training data.

[0115] In the above embodiment, text indicating the detected color, a bounding box of the detected object, and the like may be displayed in the corrected image 68 together with the object corrected with the corrected color.

[0116] In the above embodiment, an example has been described in which the corrected image 68 is transmitted from the data processing device 12 to the smart glasses 14 as color vision deficiency assistance information, but the technology of the present disclosure is not limited to this. For example, instead of the corrected image 68 as color vision deficiency assistance information, information (e.g., text data) capable of identifying the correction color and the position and size of the correction point in the visual field of the user 20 may be transmitted to the smart glasses 14, and the corrected image 68 may be generated and displayed by the smart glasses 14.

[0117] In the above embodiment, the object detection process is described by taking an example in which the object detection is performed by the YOLO method, but the technology of the present disclosure is not limited to this. For example, an object detection process other than the YOLO method may be performed. In addition, the object detection process is described by taking an example in which the object detection process is an AI method object detection process, but this is merely an example, and an object detection process by a pattern matching method may be performed. However, as described above, the object detection process by the YOLO method is desirable from the viewpoint of increasing the processing speed.

[0118] In the above embodiment, a form example in which a prompt of text data is input to the data generation model 58 has been described, but the technology of the present disclosure is not limited to this. The prompt may include image data (for example, image data showing a detected object).

[0119] In the above embodiment, the object detection process and the correction color generation process are performed separately, but the technology of the present disclosure is not limited to this. For example, a data generation model 58 having an object detection function and a correction color generation function may be used. In this case, fine tuning is performed on the data generation model 58 using a data set in which the example data is image data and the correct data is an annotation of the correction color of the detected object corresponding to the example data as teacher data.

[0120] In the above embodiment, the specific process is executed in the data processing device 12, but the technology of the present disclosure is not limited to this. For example, the specific process may be executed in the smart glasses 14. In this case, the specific process is executed in the control unit 46A of the smart glasses 14. That is, in this case, the smart glasses 14 also function as the data processing device 12.

[0121] In the above embodiment, the data processing device 12 and the smart glasses 14 communicate with each other, but the technology of the present disclosure is not limited to this. For example, the data processing device 12 and the smart glasses 14 may communicate with each other via a smart device (e.g., a smartphone or a smart watch) that is wirelessly or wired connected to the smart glasses 14. In this case, the processing in the control unit 46A may be realized by a processor installed in the smart device, and the smart glasses 14 functions as a display device.

[0122] In the above embodiment, the specification process is entirely performed in the data processing device 12, but the technology of the present disclosure is not limited to this. For example, the specification process may be performed in the control unit 46A of the smart glasses 14, and the correction color generation process may be performed in the data processing device 12.

[0123] In the above embodiment, the data generation model 58 is stored in the storage 32 of the data processing device 12, but the technology of the present disclosure is not limited to this. The data generation model 58 may be stored in a first external server separate from the data processing device 12. In this case, the data processing device 12 exchanges information with the first external server in which the data generation model 58 is stored via the communication I / F 26. Specifically, the data processing device 12 transmits a prompt generated in the processing unit 294 to the first external server, and obtains a response to the prompt output by the data generation model 58 from the first external server.

[0124] In the above embodiment, the trained model for object detection 62 is stored in the storage 32 of the data processing device 12, but the technology of the present disclosure is not limited to this. The trained model for object detection may be stored in a second external server separate from the data processing device 12. In this case, the data processing device 12 exchanges information with the second external server in which the trained model for object detection is stored via the communication I / F 26. Specifically, the processing unit 294 transmits the field of view range image 42A to the second external server, and acquires the object detection result output by the trained model for object detection 62 from the second external server.

[0125] Although the system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, the system according to the present disclosure is not necessarily implemented in a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone or the like. The method according to the present disclosure may be provided to a user in the form of SaaS (Software as a Service).

[0126] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0127] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0128] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0129] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0130] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0131] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0132] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0133] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0134] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0135] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0136] In relation to the above, the following supplementary notes are further disclosed.

[0137] <Appendix 1> an input unit for receiving an image including at least a portion of a user's field of view; A processing unit that performs specific processing using a data generation model that generates data according to an input content; an output unit that outputs color vision deficiency support information, which is information that supports visual cognition with color vision deficiency of the user, according to a processing result by the processing unit; The processing unit includes: The color vision deficiency assistance information is obtained as the processing result by using an output of the data generation model when the input content includes an instruction for generating a correction color for correcting the color of an object included in the image. Data processing device. <Appendix 2> The processing unit acquires a result of an object detection process that detects the object in the image, The input contents include information capable of identifying the object detected in the object detection process. 2. A data processing device according to claim 1. <Appendix 3> The information that can identify the object includes information that can identify the color of the detected object. 3. A data processing device according to claim 2. <Appendix 4> The input contents include instructions for generating the correction color based on the detected color of the object. 4. A data processing device according to claim 3. <Appendix 5> The information that can identify the object includes information that can identify the position and size of the object in the image. A data processing device according to any one of Supplementary Note 2 to Supplementary Note 4. <Appendix 6> The object detection process is performed using a trained model for object detection that is different from the data generation model and is capable of detecting the object in the image. A data processing device according to any one of Supplementary Note 2 to Supplementary Note 5. <Appendix 7> As the trained model for object detection, a plurality of trained models for object detection are prepared, each of which is subjected to machine learning in accordance with a plurality of scenes in which the object is detected; The trained model for object detection used in the object detection process is determined in response to a detection mode selected by the user from among the plurality of trained models for object detection. 7. A data processing device according to claim 6. <Appendix 8> The input contents include information capable of identifying the color vision characteristics of the user. A data processing device according to any one of claims 1 to 7. <Appendix 9> The input contents include an instruction for generating the correction color based on the color vision characteristics of the user. 8. A data processing device according to claim 7. <Appendix 10> The color vision deficiency assistance information includes a corrected image that is an image superimposed on the scene viewed by the user and is a partial image showing the object corrected with the corrected color output by the data generation model. 10. A data processing device according to any one of claims 1 to 9. <Appendix 11> The input contents include information that can identify the situation of the user. A data processing device according to any one of claims 1 to 10. <Appendix 12> A wearable terminal comprising a display device capable of displaying the color vision deficiency assistance information according to any one of Supplementary Note 1 to Supplementary Note 11. <Appendix 13> The wearable terminal is a glasses-type terminal. 13. The wearable terminal of claim 12. [Explanation of symbols]

[0138] 10 Data Processing System 12 Data Processing Device 14. Smart Glasses 20 users 22 Computers 24 Database 26 Communication I / F 28 processors 32 Storage 34 Bus 36 Computer 38 Reception equipment 38A Touch Panel 38B Microphone 40 Output Device 40A Display 40B Speaker 42 Camera 42A Image 42A Field of view image 44 Communication I / F 46 processors 46A Control Unit 50 Storage 52 Bus 54 Network 56 Specific Processing Program 58 Data Generation Model 60 Reception Output Program 62 Models 62A Model 62B model 64 Object detection results 64A Bounding Box 66 User color vision information 68 Corrected Images 70 User status information 72 Mode Selection Information 290 Specific Processing Department 292 Input section 294 Processing section 296 Output section

Claims

1. an input unit for receiving an image including at least a portion of a user's field of view; A processing unit that performs specific processing using a data generation model that generates data according to an input content; an output unit that outputs color vision deficiency support information, which is information that supports visual cognition with color vision deficiency of the user, according to a processing result by the processing unit; The processing unit includes: The color vision deficiency assistance information is obtained as the processing result by using an output of the data generation model when the input content includes an instruction for generating a correction color for correcting the color of an object included in the image. Data processing device.

2. The processing unit acquires a result of an object detection process that detects the object in the image, The input contents include information capable of identifying the object detected in the object detection process.

2. A data processing apparatus according to claim 1.

3. The information that can identify the object includes information that can identify the color of the detected object.

3. A data processing apparatus according to claim 2.

4. The input contents include instructions for generating the correction color based on the detected color of the object.

4. A data processing apparatus according to claim 3.

5. The information that can identify the object includes information that can identify the position and size of the object in the image.

3. A data processing apparatus according to claim 2.

6. The object detection process is performed using a trained model for object detection that is different from the data generation model and is capable of detecting the object in the image.

3. A data processing apparatus according to claim 2.

7. As the trained model for object detection, a plurality of trained models for object detection are prepared, each of which is subjected to machine learning in accordance with a plurality of scenes in which the object is detected; The trained model for object detection used in the object detection process is determined in response to a detection mode selected by the user from among the plurality of trained models for object detection.

7. A data processing apparatus according to claim 6.

8. The input contents include information capable of identifying the color vision characteristics of the user.

2. A data processing apparatus according to claim 1.

9. The input contents include an instruction for generating the correction color based on the color vision characteristics of the user.

9. A data processing apparatus according to claim 8.

10. The color vision deficiency assistance information includes a corrected image that is an image superimposed on the scene viewed by the user and is a partial image showing the object corrected with the corrected color output by the data generation model.

2. A data processing apparatus according to claim 1.

11. The input contents include information that can identify the situation of the user.

2. A data processing apparatus according to claim 1.

12. A wearable terminal comprising a display device capable of displaying the color vision deficiency assistance information according to claim 1.

13. The wearable terminal is a glasses-type terminal. The wearable terminal according to claim 12.

14. accepting an image including at least a portion of a user's field of view; Performing a specific process using a data generation model that generates data according to the input content; and outputting color vision deficiency support information, which is information for supporting visual recognition with color vision deficiency of the user, according to a processing result of the identification processing; In the identification process, the color vision deficiency assistance information is acquired as a result of the process using an output of the data generation model when the input content includes an instruction statement instructing the generation of a correction color for correcting a color of an object included in the image is input. Data processing methods.

15. On the computer, accepting an image including at least a portion of a user's field of view; Performing a specific process using a data generation model that generates data according to the input content; and A process including: outputting color vision deficiency support information, which is information for supporting visual cognition of the user with color vision deficiency, according to a processing result of the identification process; In the identification process, the color vision deficiency assistance information is acquired as a result of the process using an output of the data generation model when the input content includes an instruction statement instructing the generation of a correction color for correcting a color of an object included in the image is input. A program that executes a process.

Images