Imaging display device, wearable device and imaging display system
By combining imaging units, processing units and display units in the wearable device, using AI to generate predicted image information, the image display delay problem in the prior art is solved, real-time or pseudo-real-time image display is realized, and user operation experience is improved.
Patent Information
- Application Number
- CN202010130040.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-02-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-02-28
AI Technical Summary
Existing wearable devices such as head-mounted displays or smart glasses are difficult to achieve real-time and low-latency image display when displaying external scenery. Especially when capturing and displaying moving objects, there are significant time differences, which affects the user's operations. Experience.
Using the combination of imaging units, processing units and display units, through the array arrangement of photoelectric conversion elements and light emitting elements, combined with artificial intelligence (AI) prediction image information generation and processing, the delay time between the real image and the displayed image is reduced, and the delay time between the real image and the displayed image is realized. Real-time or pseudo-real-time image display.
By generating and processing predictive image information, the delay time between image capture and display is significantly reduced, the real-time display capability of moving objects is improved, and the user's operating experience is enhanced, especially in scenes such as capturing and displaying sports.
Smart Images

Figure CN113329165B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an imaging display device, a wearable device, and an imaging display system. Background Art
[0002] Wearable devices with imaging display devices, such as head-mounted displays or smart glasses, are known. In one system of the above-mentioned wearable devices, scenery in front of the user is captured as an image by the imaging display device, and the captured image is displayed on the display device. In the above-mentioned system, even if the external scenery is viewed via the display device, the user can feel as if the user is directly viewing the external scenery.
[0003] In order to miniaturize the above-mentioned display device, Japanese Patent Application Laid-Open No. 2002-176162 discusses a technique of arranging photodiodes and electroluminescent (hereinafter, referred to as "EL") elements in a matrix state on the same substrate. Summary of the invention
[0004] According to one aspect of the present disclosure, an imaging display device includes: an imaging unit including a plurality of photoelectric conversion elements; a processing unit configured to process a signal sent from the imaging unit; and a display unit configured to display an image based on the signal sent from the processing unit, wherein the imaging unit acquires first image information at a first time, wherein the processing unit generates first predicted image information at a second time later than the first time based on the first image information, and wherein the display unit displays an image based on the first predicted image information.
[0005] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] Figure 1A is a schematic diagram illustrating an imaging display device according to a first exemplary embodiment. Figure 1B is a schematic diagram illustrating a modification of the imaging display device according to the first exemplary embodiment. Figure 1C is a schematic diagram illustrating another modification of the imaging display device according to the first exemplary embodiment. Figure 1D is a schematic diagram illustrating still another example of the imaging display device according to the first exemplary embodiment.
[0007] Figure 2A and 2B is a diagram illustrating the operation of the imaging display device according to the first exemplary embodiment.
[0008] Figure 3 is a diagram illustrating a comparative example of the operation of the imaging display device according to the first exemplary embodiment.
[0009] Figure 4 is a schematic diagram illustrating an imaging display device according to a second exemplary embodiment.
[0010] Figure 5 is a diagram illustrating the operation of the imaging display device according to the second exemplary embodiment.
[0011] Fig. 6A and Figure 6B is a diagram illustrating the operation of the imaging display device according to the third exemplary embodiment.
[0012] Figure 7 is a schematic diagram illustrating an imaging display device according to a fourth exemplary embodiment.
[0013] Figure 8 is a diagram illustrating the operation of the imaging display device according to the fifth exemplary embodiment.
[0014] Fig. 9 is a diagram illustrating the operation of the imaging display device according to the sixth exemplary embodiment.
[0015] Fig. 10A and 10B is a schematic diagram illustrating a wearable device. Fig. 10C is a cross-sectional view schematically illustrating a positional relationship between an imaging unit and a display unit. Fig. 10D and 10E is a plan view schematically illustrating a positional relationship between an imaging unit and a display unit.
[0016] Fig.11 is a schematic diagram illustrating an imaging display system.
[0017] Fig.12 is a schematic diagram illustrating an imaging display device according to a tenth exemplary embodiment.
[0018] Fig.13 is a schematic diagram illustrating the operation of the imaging display device according to the eleventh exemplary embodiment. DETAILED DESCRIPTION
[0019] Hereinafter, exemplary embodiments will be described with reference to the accompanying drawings. In the exemplary embodiments described below, description of a configuration similar to that already described in another exemplary embodiment will be omitted. In addition, the exemplary embodiments may be appropriately changed or combined.
[0020] A first exemplary embodiment will be described below. Figures 1A to 1D as well as Figure 2A and 2B The present exemplary embodiment is described. Figure 1A1 is a schematic diagram illustrating an imaging display device 100 according to the present exemplary embodiment. The imaging display device 100 includes an imaging unit 101 , a processing unit 102 , and a display unit 103 .
[0021] The imaging unit 101 includes a plurality of light receiving elements. For example, a photoelectric conversion element is used as the light receiving element. The light receiving element performs an imaging operation for acquiring image information by converting light (external information) entering from the outside into an electrical signal. Based on the image information from the imaging unit 101, the processing unit 102 generates information about an image to be captured by the imaging unit 101 in the future (hereinafter referred to as "predicted image information"). The display unit 103 includes a plurality of light emitting elements. Each of the plurality of light emitting elements converts an electrical signal into light. The display unit 103 displays (outputs) an image based on the predicted image information generated by the processing unit 102. A plurality of pixels are arranged on the imaging unit 101 and the display unit 103 in an array state. Each pixel arranged on the imaging unit 101 includes at least one light receiving element, and each pixel arranged on the display unit 103 includes at least one light emitting element. The processing unit 102 receives image information from the imaging unit 101 and outputs the predicted image information to the display unit 103. In addition, the processing unit 102 may output a control signal for performing an imaging operation and a control signal for performing a display operation to the imaging unit 101 and the display unit 103 , respectively.
[0022] Here, we will describe Figure 1A A modification example of the imaging display device 100 according to the present exemplary embodiment is shown in FIG. Figure 1B It is a graphic Figure 1A Schematic diagram of a modified example of the imaging display device 100 according to the present exemplary embodiment. In the imaging display device 120, Figure 1A The processing unit 102 shown includes an AI unit 104 on which artificial intelligence (AI) is installed. In the present exemplary embodiment, the AI unit 104 has a deep learning (deep structured learning) function. Figure 1B In the embodiment, image information captured by the imaging unit 101 is converted into predicted image information by the processing unit 102 including the AI unit 104.
[0023] Figure 1C It is a graphic Figure 1ASchematic diagram of a modification of the imaging display device 100 of the present exemplary embodiment shown in . The processing unit 102 of the imaging display device 130 communicates with the processing device 105. The processing unit 102 and the processing device 105 are connected to each other via a network. The processing device 105 is arranged outside the imaging display device 130, such as a cloud. In the imaging display device 130, the AI unit 104 is not included in the processing unit 102, but is included in the processing device 105. The processing unit 102 and the processing device 105 exchange information to generate predicted image information based on the image information. Figure 1C In the embodiment, the image information captured by the imaging unit 101 is converted into predicted image information by the processing unit 102 which has acquired the information from the processing device 105. In this way, the imaging display apparatus 130 can generate predicted image information using information stored in an external device.
[0024] Figure 1D It is a graphic Figure 1A Schematic diagram of a modified example of the imaging display device 100 of the present exemplary embodiment shown in . The processing unit 102 of the imaging display device 140 includes an AI unit 104. The processing unit 102 communicates with a processing device 106, and the processing device 106 also communicates with another processing device 105. The processing device 106 is arranged on the cloud and stores data. The processing device 105 including the AI unit 104 is arranged separately from the imaging display device 140 and the processing device 106. The processing unit 102 and the processing device 106, and the processing device 106 and the processing device 105 are respectively connected to each other via a network. Figure 1D In the embodiment, the processing unit 102 receives the setting information stored in the processing device 106 and generates the predicted image information based on the setting information. The setting information includes basic information about the environment or the target object, and various values for generating the predicted image information. In addition, the processing unit 102 sends a plurality of pieces of information including the image information from the imaging unit 101 to the processing device 106. The plurality of pieces of information are sent to the processing device 105 via the processing device 106. The processing device 105 generates various values for generating the predicted image information based on the received plurality of pieces of information, and sends the generated values to the processing device 106. The processing device 106 updates the basic information and values stored therein, and retains the updated information and various values as new information. As described above, the imaging display device 140 can generate the predicted image information using the information stored in the external device.
[0025] exist Figure 1AIn the embodiment, the processing unit 102 predicts image information to be captured by the imaging unit 101 in the future based on the image information acquired by the imaging unit 101, and sends the image information to the display unit 103 as predicted image information. In addition, the processing unit 102 may process other types of information together with the image information, such as temperature / humidity information, acceleration information, and pressure information. Figure 1B The processing unit 102, Figure 1C The processing unit 102 and the processing device 105 and Figure 1D The processing unit 102 and the processing devices 105 and 106 in the embodiment can also be said to be the same.
[0026] Then, reference will be made to Figure 2A and 2B The operation of the imaging display device 100 of the present exemplary embodiment is described. Figure 2A and 2B 1 is a diagram illustrating the operation of the imaging display device 100 of the present exemplary embodiment and the relationship between image information and predicted image information about one frame at a time. Figure 2A and 2B , image information at time Tn is expressed as “An”, and future image information (predicted image information) processed by the processing unit 102 is expressed as “Bn”.
[0027] Reference Figure 2A The operation of the imaging display device 100 according to the present exemplary embodiment is described. In this operation, the imaging unit 101 performs an imaging operation to -2 Get image information A -2 , at time T -1 Get image information A -1 , obtaining image information A0 at time T0 and +1 Get image information A +1 Next, the processing unit 102 respectively processes the image information A based on the input image information A. -1 , A0 and A +1 To generate predicted image information B0, B +1 and B +2 Then, the processing unit 102 converts the predicted image information B0, B +1 and B +2 The display unit 103 performs a process for displaying an image based on the predicted image information B0 at time T0, an image based on the predicted image information B0 at time T +1 The predicted image information B +1 The image and the +2 The predicted image information B +2 The display operation of the image.
[0028] In other words, the imaging unit 101 is at a certain time T -1 A is used to obtain image information -1 The imaging operation is performed and at a certain time T -1 At the later time T0, the process for acquiring the image information A is performed. -1 The display unit 103 performs a display operation at time T0 for displaying the image based on the image information A -1 The image of the predicted image information B0 is generated. +1 At this point, the imaging unit 101 performs an imaging operation to obtain image information A different from the image information A0. +1 Then, the display unit 103 performs a display operation for displaying the predicted image information B generated from the image information A0. +1 of the image.
[0029] Here, we will refer to Figure 3 A comparative example is described. In an imaging display device that does not have the processing unit 102 described in the present exemplary embodiment, at time T -1 Image information A captured by the imaging unit 101 -1 is displayed by the display unit 103 at time T0.
[0030] The description will be displayed Figure 2A The configuration of the predicted image information of the present exemplary embodiment is different from that displayed as is by Figure 3 If the current time is time T0, then image information A0 corresponds to a real phenomenon (i.e., an actual image) at that time. Information used as a source of an image to be displayed by the display unit 103 of the present exemplary embodiment is predicted image information B0. If the image information captured by the imaging unit 101 is used as it is, then information used as a source of an image displayed by the display unit 103 is image information A0. -1 Here, the amount of change in image information is expressed as “(image information A -1 -image information A0)≥(predicted image information B0-image information A0)". Thus, with the configuration described in the present exemplary embodiment, it is possible to acquire an imaging display device that performs reduction of the difference between a real phenomenon and a displayed image.
[0031] The timing for displaying predicted image information of the present exemplary embodiment will be described. The processing unit 102 of the present exemplary embodiment generates predicted image information to reduce the delay time between the image information captured by the imaging unit 101 at a certain time and the image to be displayed by the display unit 103. It is desirable to set the timing for displaying the predicted image information as follows.
[0032] First, it is assumed that the imaging unit 101 captures an image at an optional time Tn. The processing unit 102 generates predicted image information based on the image information acquired at time Tn. The time when the display unit 103 displays an image based on the predicted image information generated about time Tn is expressed as time Tm. Here, the difference ΔT between the imaging timing and the display timing can be expressed by Formula 1.
[0033] ΔT=Tn–Tm Formula 1
[0034] The display frame rate DFR (fps: frames per second) represents the number of images displayed per second by the display unit 103. The imaging display device is controlled so that the difference ΔT satisfies Formula 2. More preferably, the imaging display device is controlled so that the difference ΔT satisfies Formula 3.
[0035] -2 / DFR≤ΔT≤2 / DFR Formula 2
[0036] -1 / DFR≤ΔT≤1 / DFR Formula 3
[0037] For example, when the display frame rate is 240 fps, the time it takes to display an image (one frame) after it is captured is approximately 4×10 -3 Seconds. Therefore, the difference ΔT is expressed as follows.
[0038] -4×10 -3 ≤ΔT≤4×10 -3 Formula 4
[0039] By displaying an image based on the predicted image information at the above timing, a moving image with a small amount of delay time between the actual image and the displayed image can be displayed. This moving image display can be called "real-time display". Thus, in this exemplary embodiment, real-time display, or more precisely, pseudo-real-time display, can be performed. In addition to still images, the present disclosure is also effectively applied to moving images.
[0040] In addition, in addition to displaying images at the above-mentioned timing, the above-mentioned timing difference can also be used to generate predicted image information. The image information captured by the imaging unit 101 at an optional time is expressed as "An". The image information displayed by the display unit 103 at the same time is expressed as "Dn". Here, the difference between the image information An and the image information Dn (that is, the time difference between the image information An and the image information Dn) can be expressed as ΔA=Dn-An. Figure 2A In an exemplary embodiment of , the image information Dn is equal to the image information Bn (Dn=Bn). In other words, the time difference between the image information captured by the imaging unit 101 at a certain time (i.e., the real phenomenon (actual image) at that time) and the image information displayed by the display unit 103 may be ±4×10-3 seconds. When the time difference between image information is ±4×10 -3 seconds, the image displayed by the display unit 103 is delayed by 4×10 -3 seconds delayed image, or 4×10 seconds after the actual image at a certain time -3 The predicted image information is expected to be generated under the above conditions. In addition, for example, the image information An can be compared with the image information Dn by using the original data of the image information An and the image information Dn. Then, when the root mean square of the difference is calculated, the image information Dn can fall within ±4×10 -3 The processing unit 102 sets various parameters for generating the next predicted image information by using this difference information.
[0041] like Figure 3 As shown in FIG. 1 , when the image information captured by the imaging unit 101 is displayed on the display unit 103, a delay time occurs. In particular, when additional image processing is performed, the delay time becomes 100×10 -3 However, by generating the predicted image information of the present exemplary embodiment, an image having no time difference from an actual image can be displayed.
[0042] As examples of additional image processing, dark field image processing for increasing the brightness of dark images, image magnification processing for magnifying and displaying small objects, and thermal display processing for displaying thermal images are given. Through the processing according to this exemplary embodiment, real-time display can be performed even if additional time is required to perform the above-mentioned image processing.
[0043] Next, we will describe Figure 2B In this operation, the imaging unit 101 performs an imaging operation for -2 Get image information A -2 , at time T -1 Get image information A -1 , obtaining image information A0 at time T0 and +1 Get image information A +1 The processing unit 102 is based on the input image information A -1 , A0 and A +1 Generate predicted image information B +1 , B +2 and B +3 Then, the processing unit 102 converts the predicted image information B +1 , B +2 and B +3 The display unit 103 performs a display operation for displaying the image based on the predicted image information B at time T0.+1 The image at time T +1 Display based on predicted image information B +2 The image and the +2 Display based on predicted image information B +3 In other words, the processing unit 102 predicts the image information to be captured at time T0, and the display unit 103 displays the image based on the predicted image information at time T0. In this way, information about a future time later than the imaging time can be displayed at the imaging time. By continuously repeating the above operation, an image of a future time later than the actual image is continuously displayed, that is, the image can be displayed as a video image.
[0044] The image information used as a source of predicted image information will be described. Figure 2A In the example, based on the image information A -1 Generate predicted image information B0. Figure 2B In the example, based on the image information A -1 Generate predicted image information B +1 In other words, one piece of predicted image information is generated based on one piece of image information. However, one piece of predicted image information may be generated based on two or more pieces of image information. For example, Figure 2A In the example, we can use the image information A -2 and A -1 To generate the predicted image information B0, and Figure 2B In the example, we can use the image information A -2 and A -1 To generate the predicted image information B +1 Thus, predicted image information can be generated by using at least one piece of image information.
[0045] The frame rate of the present exemplary embodiment will be described. First, the number of pieces of image information acquired by the imaging unit 101 per second is defined as the imaging frame rate SFR (fps). Then, as described above, the number of pieces of image information displayed by the display unit 103 per second is defined as the display frame rate DFR (fps). At this time, in the present exemplary embodiment, Figure 2A and 2B The relationship between the frame rates in is expressed as "SFR=DFR". However, the imaging frame rate and the display frame rate may be different from each other. Specifically, it is desirable that the frame rate satisfies the condition "SFR≥DFR" because predicted image information can be generated from a plurality of pieces of captured image information.
[0046] Next, the configuration of the imaging display device 100 will be described. Examples of the photoelectric conversion element included in the imaging unit 101 include a photodiode, a photogate, and a photoelectric conversion film. For example, silicon, germanium, indium, gallium, and arsenic can be used as materials for the photodiode and the photogate. As examples of photodiodes, a positive-negative (PN) junction photodiode, a positive-intrinsic-negative (PIN) photodiode, and an avalanche photodiode can be given.
[0047] For example, a complementary metal oxide semiconductor (CMOS) image sensor may be used as the imaging unit 101, and the CMOS image sensor may be a front illumination type or a back illumination type. In addition, the CMOS image sensor may have a structure in which a semiconductor substrate having a photodiode disposed thereon and a semiconductor substrate having a scanning circuit and a control circuit disposed thereon are stacked on each other.
[0048] In addition, the material of the photoelectric conversion film can be an organic material or an inorganic material. The organic photoelectric conversion film has a structure arranged at a position between a pair of electrodes, and the structure has at least one organic layer for performing photoelectric conversion. The organic photoelectric conversion film can also have a structure having a plurality of organic layers stacked and arranged at a position between a pair of electrodes. The organic layer can be made of a single material or a plurality of materials mixed together. In addition, the organic layer can be formed by a vacuum vapor deposition process or a coating process. For example, a quantum dot-type photoelectric conversion film using a quantum dot thin film layer containing fine semiconductor crystals instead of an organic layer, or a perovskite-type photoelectric conversion film including a photoelectric conversion layer composed of a transition metal oxide having a perovskite structure is used as an inorganic photoelectric conversion film.
[0049] The display unit 103 includes a plurality of light-emitting elements. The light-emitting element may be a liquid crystal display (LCD), an inorganic light-emitting diode (LED), an organic LED (OLED) or a quantum dot LED (QLED). For example, materials such as aluminum, gallium, arsenic, phosphorus, indium, nitrogen, selenium, zinc, diamond, zinc oxide and / or perovskite semiconductors are used for inorganic LEDs. The inorganic LED with a PN junction structure formed by using the above-mentioned materials emits light with energy (wavelength) corresponding to the band gap between the above-mentioned materials. For example, an organic LED includes a light-emitting layer, which includes at least one organic light-emitting material arranged at a position between a pair of electrodes. The organic LED may include a plurality of light-emitting layers, and may have a structure in which a plurality of organic layers are stacked one on top of another. The light-emitting layer may be made of a single material or a plurality of materials mixed together. The light emitted from the light-emitting layer may be fluorescence or phosphorescence, and the light may be monochromatic light (e.g., blue, green or red light) or white light. In addition, the organic layer may be formed by vacuum vapor deposition or coating.
[0050] In addition, the imaging display device may have a structure in which at least three chips of the imaging unit 101 , the processing unit 102 , and the display unit 103 are stacked and electrically connected to each other through a semiconductor process.
[0051] In the case where the imaging display device 100 of the present exemplary embodiment is used as a wearable device, it is desirable that the amount of data processed by the processing unit 102 is as small as possible. This is because the wearable device needs to be as small in size, weight, and thickness as possible, and if the data processing load is small, the chip size of the processing unit 102 can be further reduced. In order to reduce the data processing load, AI processing can be performed by, for example, Figure 1C Or another device shown in 1D (for example, a device provided on the cloud). In addition, a method for reducing the resolution of the portion outside the line of sight area, a method for processing the portion outside the line of sight area as a still image, or a method for processing the portion outside the line of sight area in monochrome rather than color can be used as a method of reducing the amount of processing.
[0052] If it takes a long time to capture and display an image of a real phenomenon such as a landscape, a difference may occur between the real phenomenon and the displayed image. For example, if there is a difference between the real phenomenon and the displayed image, the user cannot perform an operation for capturing a moving object. However, according to the present exemplary embodiment, an imaging display device that reduces the time difference between the real phenomenon and the displayed image can be provided. Therefore, an operation for capturing a moving object can be performed.
[0053] Hereinafter, a second exemplary embodiment will be described. Figure 4 and Figure 5 The present exemplary embodiment is described. Figure 4 is a schematic diagram illustrating an imaging display device 400 according to the present exemplary embodiment. Figure 1C Similar to the imaging display device 130 shown in FIG. 1 , the imaging display device 400 includes an imaging unit 101, a processing unit 102, a display unit 103, and a processing device 105. The imaging display device 400 also includes a detection unit 107 for detecting environmental information. In the imaging display device 400, each of the processing unit 102 and the processing device 105 includes an AI unit 104.
[0054] The detection unit 107 includes at least one sensor. The sensor can detect at least one piece of environmental information. As examples of environmental information, information about atmospheric temperature, water temperature, humidity, atmospheric pressure, water pressure, and brightness is given. In addition, the detection unit 107 can also obtain physical information such as the acceleration of an object captured by the imaging unit 101. In this exemplary embodiment, although the detection unit 107 is built in the imaging display device 400, the detection unit 107 can be arranged outside.
[0055] For example, the AI unit 104 with a deep learning function generates predicted image information based on the image information acquired from the imaging unit 101 and the environmental information acquired from the detection unit 107. At this time, predicted image information about a time taking into account the time taken for the processing unit 102 to perform processing is generated. In other words, if an image is captured at a certain time, predicted image information related to a future time is generated, and the time required to perform processing is added to the certain time in addition to the time taken to capture and display the image. The details of the operation are similar to those described in the first exemplary embodiment.
[0056] Subsequently, the processing performed by the AI unit 104 will be described taking a scene in which people are playing baseball as an example. Figure 5 is a diagram illustrating the operation of the imaging display device of the present exemplary embodiment. Figure 5 The diagram shows the -3 , T -2 , T -1 Examples of four types of images captured at and T0.
[0057] Figure 5 The actual image (under illumination) in FIG. 1 illustrates the actual image. Here, the actual image refers to the image at time T -3 , T -2 , T -1 The actual image (under illumination) illustrates the image captured by the imaging unit 101 at each time T0 when a person plays baseball in a bright environment (under illumination). -3 , T -2 , T -1 and the actual image captured at each time in T0. Because the pitcher and the ball can be clearly identified, the batter can hit and return the ball at time T0.
[0058] The actual image (in the dark) shows the actual image. Here, the actual image refers to the image at time T -3 , T -2 , T -1 The actual image (in the dark) illustrates the image captured by the imaging unit 101 at each time T0 and T1. The actual image (in the dark) illustrates the image captured by the imaging unit 101 at each time T1 when people play baseball in a dark environment (in the dark). -3 , T -2 , T -1 The actual image captured at each time in T0 and T1. Since the ball cannot be seen by the naked eye in such a dark environment, the batsman cannot hit and return the ball at time T0. Therefore, although the ball is not visible to the naked eye in such a dark environment, the batsman cannot hit and return the ball at time T1. -3 , T -2 and T -1The movement and position of the pitcher and ball are the same in the actual image under light and in the actual image in darkness, but different results are obtained at time T0.
[0059] The image of the comparative example illustrates a situation where the imaging display device of the comparative example is used when people play baseball in the dark. The image of the comparative example represents an image obtained by additionally performing image processing on an actual image captured in the dark and displayed on the imaging display device. Even in a state shown in an actual image captured in the dark (where an object cannot be seen by the naked eye), an image that is the same as the actual image captured under light can be displayed by performing additional processing on the captured image information. The additional image processing is a process for increasing the brightness of an image captured in the dark. By performing the above-mentioned processing, the batter can see the ball. However, due to the delay time, at time T -1 The position of the ball at time T0 is different from the position in the actual image. Therefore, the batsman cannot hit and return the ball at time T0.
[0060] In the image of the present exemplary embodiment, an exemplary embodiment of the imaging display device of the present exemplary embodiment will be described when people play baseball in the dark. The image of the present exemplary embodiment is an image displayed on the imaging display device 100 after additional image processing is performed on the actual image captured in the dark. Even in the state shown in the actual image captured in the dark (where the object cannot be seen by the naked eye), it is possible to display the same image as the actual image captured under light by performing additional processing on the captured image information. In addition, as described in the imaging display device 100 of the present exemplary embodiment, an image using predicted image information without a delay time can be displayed. Thus, the batter can hit and return the ball at time T0, as if the batter is playing baseball under light. At time T -3 , T -2 , T -1 At and T0, the movement and position of the pitcher and the ball in the image of the present exemplary embodiment are substantially the same as the movement and position of the pitcher and the ball in the actual image captured under illumination.
[0061] As in this exemplary embodiment, an image can be displayed in real time by an imaging display device using predicted image information. Therefore, the position of a moving object such as a ball can be accurately identified. In addition, by using predicted image information, an image can be displayed in real time even if additional processing is performed on the captured image information. The imaging display device according to this exemplary embodiment is ideal for capturing and displaying the movement of a moving object in a sports game such as baseball.
[0062] In addition, Figure 5In the example shown, the detection unit 107 detects information about the wind direction and wind speed, and outputs the detected information to the processing unit 102. The processing unit 102 predicts the speed or route of the ball based on the above information and generates predicted image information about the ball. In addition, the detection unit 107 can determine whether additional image processing is to be performed by the processing unit 102 by detecting brightness information.
[0063] In addition, in the imaging display device 400 of the present exemplary embodiment, it is expected that the imaging frame rate SFR is greater than the display frame rate DFR. For example, the imaging frame rate SFR and the display frame rate DFR can be set to 500fps and 60fps, respectively. Because the imaging frame rate SFR is greater than the display frame rate DFR (SFR>DFR), a piece of predicted image information can be generated based on multiple pieces of image information. Therefore, the consistency rate between the displayed image and the actual image is increased, so that the movement of the moving object can be accurately displayed.
[0064] In addition, send to Figure 1D The various types of information in the external processing device 105 include Figure 4 Various types of information acquired by the detection unit 107 of the present exemplary embodiment shown in . For example, batter's batting data, data on the pitcher's batting form, weather information, and user (batter) information are basic information stored in the processing device 106.
[0065] Hereinafter, a third exemplary embodiment will be described. In this exemplary embodiment, another operation of the imaging display device 400 described in the second exemplary embodiment will be described. Fig. 6A and 6B is a diagram illustrating the operation of the imaging display device of the present exemplary embodiment. Fig. 6A The actual image is shown, and Figure 6B An image based on predicted image information is illustrated.
[0066] Fig. 6A The actual image shown in can be referred to as image information 600 acquired by the imaging unit 101 at a certain time. The processing unit 102 detects a moving object 602 by using the image information captured before a certain time and at least two pieces of image information included in the image information 600. A characteristic stationary object 601 in the scenery can be detected at the same time. The characteristic stationary object 601 can be identified by the line of sight detection described below. Line of sight detection refers to eye tracking. In Fig. 6A In the image 600, for example, two trees are characteristic stationary objects 601, and a train is a moving object 602. The subsequent image processing method will be described.
[0067] One of the image processing methods is a method in which the AI unit 104 partially generates predicted image information only about a portion of the image information determined to be the moving object 602. Then, because the actual image of the stationary object 601 is unlikely to be changed, predicted image information based on the image information about the stationary object 601 is not generated. The predicted image information 610 generated by the above method is Figure 6B For the purpose of illustration, Fig. 6A The moving object 602 in Figure 6B In the predicted image information 610, the position of the moving object 612 is different from the position of the moving object 602, and the position of the stationary object 611 is not changed from the position of the stationary object 601 (not shown). Through the above-mentioned processing, the load of the processing unit 102 can be reduced while displaying the image in real time.
[0068] Another image processing method is a method of further performing additional image processing only on the portion determined as the moving object 602. In the additional image processing, the resolution of the portion determined as the moving object 602 is increased and / or refined, and the resolution of each portion determined as the stationary object 601 is reduced. The high-resolution moving object 612 and the low-resolution stationary object 611 are displayed on the display unit 103. Since the processing is changed depending on the portion, the load on the processing unit 102 can be reduced. In addition, since the moving object 612 is displayed at a high resolution and the stationary object 611 is displayed at a low resolution, a natural image close to an image seen by the human eye can be provided.
[0069] Another image processing method is a method in which, if there is time before a signal is output to the display unit 103 after the predicted image information is generated about the moving object 602, the predicted image information is generated about the part of the stationary object in the surroundings of the moving object 602. With this method, a higher-precision image can be displayed in real time.
[0070] As described above, the processing unit 102 performs moving object detection on the image data received from the imaging unit 101 and changes the processing method depending on the part. With this processing method, it is possible to display an image with high quality while reducing the load on the processing unit 102.
[0071] In addition, a method other than the method using two or more pieces of image information is provided as a method for detecting a stationary object and a moving object. For example, a moving object detection unit may be provided as the detection unit 107. The moving object detection unit may include a ranging sensor. The number of stationary objects 601 and moving objects 602 to be detected by the moving object detection unit is not limited.
[0072] Hereinafter, a fourth exemplary embodiment will be described. Figure 7 and 8 The imaging display device of the present exemplary embodiment is described. Figure 7 1 is a schematic diagram illustrating an imaging display device 700 of the present exemplary embodiment. The imaging display device 700 includes a sight line detection unit 108 in place of Figure 4 The imaging display device 400 shown in FIG. Figure 7 In the embodiment, the sight line detection unit 108 is built into the imaging display device 700. Alternatively, the sight line detection unit 108 may be arranged externally.
[0073] exist Figure 7 , the image information captured by the imaging unit 101 is output to the processing unit 102. At the same time, the line of sight information acquired by the line of sight detection unit 108 is output to the processing unit 102. The processing unit 102 generates predicted image information through the AI unit 104. In addition, the processing unit 102 generates predicted line of sight information by predicting the movement and position of the line of sight in the future based on the line of sight information. Then, based on the predicted line of sight information, the processing unit 102 performs additional image processing to increase the resolution of the part where the line of sight exists and reduce the resolution of the part other than the part where the line of sight exists, and generates the final predicted image information. While performing real-time display, the display unit 103 can display the part where the predicted line of sight exists with high resolution, and display the part other than the part where the predicted line of sight exists with low resolution.
[0074] Reference Figure 8 Describe the operation in detail. Figure 8 is a diagram illustrating the operation of the imaging display device of the present exemplary embodiment. Figure 5 resemblance, Figure 8 The diagram shows a scene of people playing baseball. Figure 8 The diagram shows the -3 , T -2 , T -1 and examples of the three types of images captured at T0.
[0075] The actual image is the image at time T -3 , T -2 , T -1and images captured at each time in T0. The batter hits and returns the ball at time T0. The descriptive image schematically illustrates the line of sight (line of sight area) on the actual image. The line of sight is detected by the line of sight detection unit 108. In the descriptive image, the line of sight is adjusted to the ball and moves as the ball moves. The image of this exemplary embodiment is an image obtained using the predicted line of sight information of this exemplary embodiment. This image is based on the predicted image information, and image processing for increasing the resolution of the part where the line of sight exists and reducing the resolution of the part other than the part where the line of sight exists is additionally performed on the predicted image information. The ball part is displayed at a high resolution, and the pitcher part is displayed at a low resolution. Through the above processing, a high-quality image can be displayed in real time while reducing the load of the processing unit 102.
[0076] Although the imaging display device 700 includes the sight line detection unit 108 instead of Figure 4 , but the imaging display device 700 may include both the detection unit 107 and the sight line detection unit 108. Its configuration may be appropriately changed. In addition, the sight line detection unit 108 may adopt an optional method, such as a method for detecting the position of the iris of the eye or a method using corneal reflection by emitting infrared light.
[0077] Hereinafter, a fifth exemplary embodiment will be described. Fig. 9 A processing method to be performed when the visual line detection unit 108 of the fourth exemplary embodiment detects a plurality of visual line areas is described. Fig. 9 is a diagram illustrating the operation of the imaging display device of the present exemplary embodiment. Figure 8 resemblance, Fig. 9 is a diagram illustrating a scene in which people are playing baseball. Fig. 9 The diagram shows the -3 , T -2 and T -1 Three examples of captured images. In the present exemplary embodiment, predicted image information is generated in which a plurality of sight lines are weighted and processed as a resolution based on the weighting. The display unit 103 displays the image of the portion corresponding to the sight line area at the resolution based on the weighting, and displays the image of the portion other than the sight line area at a low resolution.
[0078] Reference Fig. 9 The actual image is at time T -3 , T -2 and T -1 Each actual image includes the pitcher and runner on first base, and the batter is about to hit the ball at time T. -1 Hit and return the ball.
[0079] Fig. 9 The descriptive image of schematically illustrates the sight line areas on the actual image. A weighting value (%) is indicated for each sight line area. -3 , the batter's sight is adjusted to the ball thrown by the pitcher and the base runner. Here, the sight area of the ball is weighted 60%, and the sight area of the base runner is weighted 40%. At time T -2 , adjust the batter's sight to the ball. Therefore, the sight areas of the ball, base runners, and pitcher are weighted 90%, 8%, and 2%, respectively. At time T -1 , the batter's sight line is mainly adjusted to the ball. The sight lines of the ball, the base runner, and the pitcher are weighted by 98%, 1%, and 1%, respectively. This weighting is performed by the processing unit 102, and the weighted value can be determined based on the movement of the sight line detected by the sight line detection unit 108. Alternatively, the determination can be performed by another AI unit 104.
[0080] Fig. 9 The image of the present exemplary embodiment shown in is an image using predicted line of sight information according to the present exemplary embodiment. The processing unit 102 adjusts the resolution of the portion based on the weighting of the line of sight area shown in the descriptive image, and generates predicted image information. Each value shown in the image of the present exemplary embodiment represents a ratio of resolution, and this is a value when the maximum resolution is 100%. Portions outside the line of sight area can be displayed at the minimum resolution. With this configuration, an image close to an image seen by the human eye can be displayed, and the processing load of the processing unit 102 can also be reduced.
[0081] Hereinafter, a sixth exemplary embodiment will be described. The imaging display device of this exemplary embodiment can display an image using light other than visible light (near infrared light, infrared light, and ultraviolet light). Figure 4 In the imaging display device 400 shown in , the imaging unit 101 includes a photoelectric conversion element that performs detection of the visible light region and a photoelectric conversion element that performs detection of light in a wavelength band outside the visible light region. For example, the imaging unit 101 includes at least two area sensors. A photoelectric conversion element for visible light is arranged on one of the two area sensors, and a photoelectric conversion element for invisible light is arranged on the other area sensor. Alternatively, the imaging unit 101 includes one area sensor. The area sensor includes at least one photoelectric conversion element for visible light and one photoelectric conversion element for invisible light.
[0082] Through the above-mentioned imaging unit 101, in addition to the image information about the visible light region, the image signal of the invisible light region including the near-infrared light region can also be obtained. Based on the above-mentioned image information, the processing unit 102 generates predicted image information about a visible light region. With this configuration, even in a state of low sensitivity in the visible light region, an image with improved and / or refined sensitivity can be displayed. In other words, the imaging display device of this exemplary embodiment also displays images that cannot be seen by the human eye in real time. The above-mentioned imaging display device of this exemplary embodiment can be applied to, for example, night vision equipment, surveillance equipment, binoculars, telescopes, and medical detection equipment.
[0083] Hereinafter, the seventh exemplary embodiment will be described. In the above exemplary embodiments, predicted image information is generated by AI processing using a deep learning function. In this exemplary embodiment, a trained model established by machine learning is used. Data is collected from experts, and a trained model is established based on the collected data. Then, an imaging display device in which predicted image information is generated based on the data collected from experts is applied to non-experts. For example, data is acquired from professional athletes, and a trained model is established based on the acquired data. Then, this trained model is used by an AI unit to generate predicted image information. By using the imaging display device of this exemplary embodiment, non-professional athletes can virtually experience the sight or attention of professional athletes, thereby improving and / or refining the athletic ability of non-professional athletes in a shorter period of time. This exemplary embodiment is also applicable to fields where it is desired to inherit the expertise of experts. For example, this exemplary embodiment can be applied to occupations in various fields requiring professional skills, such as pilots, doctors, traditional craftsmen, and security services.
[0084] Hereinafter, an eighth exemplary embodiment will be described. Figures 10A to 10E An application example of the imaging display device according to each of the above exemplary embodiments when applied to a wearable device is described. The imaging display device can be applied to a wearable device such as smart glasses, a head mounted display (HMD), or a smart contact lens.
[0085] Fig. 10Ais a schematic diagram illustrating smart glasses 1000. Smart glasses 1000 are also referred to as glasses-type imaging display devices or glasses. Smart glasses 1000 include a glasses frame and an imaging display device according to the above exemplary embodiment. Specifically, smart glasses 1000 include at least two imaging units 1001, a processing unit 1002, and a display unit 1003. The two imaging units 1001 are arranged on the sides of the glasses frame, and the processing unit 1002 is accommodated in the temples of the glasses. The display unit 1003 is arranged at an optional position depending on the display form and may be included in the lens 1011. In any case, the display unit 1003 displays an image on the lens 1011. The processing unit 1002 may include an AI unit. Smart glasses 1000 may include an external interface so that the processing unit 1002 can exchange data with an external AI unit.
[0086] Fig. 10A The smart glasses 1000 in the embodiment may include two imaging display devices for the right eye and the left eye, respectively. In this case, the timing for capturing and displaying images may be optionally set in the corresponding imaging display devices for the right eye and the left eye. Specifically, images may be captured at the same time and displayed at different times, or images may be captured at different times and displayed at the same time.
[0087] Fig. 10B 1 is a schematic diagram illustrating a smart contact lens 1020. The smart contact lens 1020 is also referred to as a contact lens-type imaging display device or a contact lens. The smart contact lens 1020 includes an imaging display device 1021 and a control device 1022. The control device 1022 serves as a power supply unit for supplying power to the imaging display device 1021. The control device 1022 includes an AI unit and supports a processing unit of the imaging display device 1021. In addition, the AI unit may be arranged on a terminal different from the smart contact lens 1020. It is desirable to arrange an optical system for collecting light to the imaging display device 1021 on the smart contact lens 1020. The power supply unit includes an interface for connecting a power supply to an external part. The power supply unit may be connected to an external part and charged via a wired connection or a wireless connection.
[0088] Transparent materials are used as Fig. 10A Lenses 1011 and Fig. 10B The base material of the smart contact lens 1020 in the embodiment of the present invention is used, and the display unit of the imaging display device projects the display image on the transparent lens part. At this time, it can be displayed on, for example, Figure 2BAn image based on predicted image information at a future time later than the display time shown in , so that the user can see both the actual image and the predicted image in the future. Because image information at a time slightly later than the current time can be displayed in real time, for example, a user playing in the outfield of a baseball game can pre-occupy a defensive position by moving in the direction of the ball hit by the batter. At this time, since the user can see both the actual image and the predicted image, the user can achieve a higher level of athletic performance than the user's actual athletic performance level. In addition, the imaging display device can freely adjust the timing for displaying the acquired image information. With this configuration, an operation suitable for the user can be selected.
[0089] like Fig. 10A As shown in , the imaging unit 1001 and the display unit 1003 may be arranged at different positions, or may be stacked and arranged in a line of sight. Fig. 10C 1001 and 1003. Fig. 10D 1001 is a schematic plan view of the imaging unit 1001 and the display unit 1003 as viewed from the imaging unit 1001 side. Fig.10E 1001 and the display unit 1003 are schematically shown in plan view when viewed from the display unit 1003 side. Fig. 10D The centroid of the imaging area 1031 where the pixels of the imaging unit 1001 are arranged is illustrated. Fig.10E The figure shows the centroid of the display area 1033 where the pixels of the display unit 1003 are arranged. Fig. 10C As shown in , it is desirable to arrange the imaging unit 1001 and the display unit 1003 in a state where the line segment A passes through the two centroids. With this configuration, it is possible to reduce variations in the wearable device caused by a positional difference between captured image information and a displayed image.
[0090] Hereinafter, a ninth exemplary embodiment will be described. In this exemplary embodiment, an imaging display system will be described. Fig.11 1 is a schematic diagram illustrating an imaging display system 1100. The imaging display system 1100 of this exemplary embodiment includes a plurality of imaging display devices 1101 and at least one control device 1102. The imaging display device 1101 may be an imaging display device described in any of the exemplary embodiments, for example, Figure 1A The imaging display device 100 shown in FIG.
[0091] The plurality of imaging display devices 1101 may receive signals from the control device 1102 and transmit signals to the control device 1102. Each of the imaging display devices 1101 and the control device 1102 includes an external interface unit for performing wired or wireless communication. The control device 1102 receives signals from the plurality of imaging display devices 1101 and outputs signals for controlling the plurality of imaging display devices 1101. The control device 1102 may include a portion of the functions of the processing unit 102 of the imaging display device 1101. The imaging display system 1100 may further include a data storage unit, a control unit, and a processing unit. For example, the imaging display system 1100 may include Figure 1D In this case, the control device 1102 can communicate with the processing device 105 or 106.
[0092] For example, the imaging display system 1100 of the present exemplary embodiment can display an image on a single display unit 103 by using a plurality of pieces of image information acquired from a plurality of corresponding imaging units 101 included in a plurality of imaging display devices 100. Specifically, for example, when a plurality of users use the corresponding imaging display devices 100, a plurality of pieces of image information are acquired from the plurality of imaging display devices 100, and the image can be displayed in real time on another imaging display device 100 used by a user different from the plurality of users. For example, at least one spectator can simultaneously see in real time the sight line images of professional athletes playing in the same stadium.
[0093] In addition, images can be displayed on multiple display units 103 using image information acquired from a single imaging unit 101. Specifically, image information acquired from one imaging display device 100 can be displayed in real time on multiple imaging display devices 100 different from the imaging display device from which the image information was acquired. For example, a line of sight image of a professional athlete can be viewed simultaneously by multiple viewers in real time.
[0094] As described above, by virtually experiencing the sight lines of professional athletes, the viewer can view images with a sense of telepresence, which makes the viewer feel as if the viewer himself is in the stadium.
[0095] In addition, this system allows the respective display devices to perform imaging operations or display operations at different intervals. In addition, in this system, image information and various types of information acquired by multiple imaging display devices can be shared and used to create predicted image information on multiple imaging display devices.
[0096] In the following, reference will be made to Fig.12 A tenth exemplary embodiment is described. Fig.12 is a schematic diagram illustrating an imaging display device 800 of the present exemplary embodiment. Fig.12The imaging display device 800 shown in FIG. Figure 1C The imaging display device 130 in FIG. The same reference numerals are applied to the same elements, and description thereof will be omitted. The imaging display device 800 includes an imaging unit 101, a processing unit 102, a display unit 103, and a processing device 105. The imaging display device 800 also includes a recording unit 109 for recording image information. In the imaging display device 800, each of the processing unit 102 and the processing device 105 includes an AI unit 104. Although the imaging display device 800 is Fig.12 The recording unit 109 is included in the processing unit 102, but the recording unit 109 may be included in the processing unit 102. The position of the recording unit 109 can be set appropriately. The information to be input to the recording unit 109 is, for example, image information from the imaging unit 101, that is, information before being converted into predicted image information. By recording the above-mentioned image information, the recording unit 109 can also obtain images other than predicted images.
[0097] The operation of the imaging display device 800 according to the present exemplary embodiment will be described. Image information from the imaging unit 101 is input to both the recording unit 109 and the processing unit 102 that generates predicted image information. The operation to be performed after the image information is input to the processing unit 102 is similar to the operation described in the first exemplary embodiment. The image information input to the recording unit 109 is not converted into a predicted image, but is directly recorded as image information acquired by the imaging unit 101. With this configuration, the time difference between the real phenomenon and the displayed image is reduced by using the predicted image information generated by the processing unit 102, and the image information acquired by the imaging unit 101 can be recorded as it is.
[0098] Hereinafter, an eleventh exemplary embodiment will be described. In the present exemplary embodiment, a method of further performing the enlargement process by the processing unit 102 and the AI unit 104 of the first exemplary embodiment will be described. Fig.13 is a diagram illustrating the operation of the imaging display device of the present exemplary embodiment. Fig. 6A , Fig.13 The external information in represents the actual image, that is, the image information acquired by the imaging unit 101.
[0099] Example 1 is a comparative example. The enlarged image of Example 1 illustrates partial image information acquired by performing enlargement processing on a part of a specific area as image information acquired by the imaging unit 101. Here, the display unit 103 displays an image based on the partial image information. As shown in Example 1, in general, when enlargement processing is performed on partial area information, the resolution of the partial area information is reduced.
[0100] Example 2 is an example according to the present exemplary embodiment. The enlarged image of Example 1 illustrates partial image information obtained by performing enlargement processing on a portion of a specific area as image information obtained by the imaging unit 101. Here, the display unit 103 displays an image based on the partial image information. In Example 2, the processing unit 102 performs enlargement processing on the one portion, and further performs resolution improvement processing on it. Resolution improvement processing is processing for improving resolution. For example, resolution improvement processing can be performed on partial pixel information by performing inter-pixel compensation processing using multiple image information. In addition, resolution improvement processing is performed by compensation processing using multiple image information or compensation processing for estimating a shape by detecting the contour of image information.
[0101] In the imaging display device of the present exemplary embodiment, a magnification process of a portion of an image based on image information may be performed by the processing unit 102 and the AI unit 104. Therefore, an image based on the magnified partial image information may be displayed on the display unit 103. The imaging display device of the present exemplary embodiment may perform a resolution improvement process on the partial image information. In addition, the image based on the magnified partial image information described in the present exemplary embodiment may be implemented simultaneously with the function for reducing the time difference between the real phenomenon and the displayed image as described in other exemplary embodiments.
[0102] Although the exemplary embodiment has been described by taking baseball as an example, the present disclosure is not limited thereto. The imaging display device of the present disclosure can reduce the time difference between the real phenomenon and the displayed image so that the user can use the device without feeling uncomfortable. As described above, according to one aspect of the present disclosure, an imaging display device that performs the reduction of the difference between the real phenomenon and the displayed image can be obtained.
[0103] The (one or more) embodiments of the present disclosure may also be implemented by a computer of a system or device, which reads and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above (one or more) embodiments, and / or the computer includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above (one or more) embodiments, and by, for example, reading and executing computer executable instructions from a storage medium to perform the functions of one or more of the above (one or more) embodiments, and / or controlling one or more circuits to perform the functions of one or more of the above (one or more) embodiments, to be implemented by a method executed by a computer of a system or device. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD) TM ), one or more of flash memory devices, memory cards, etc.
[0104] Although the present disclosure has been described with reference to exemplary embodiments, it is to be understood that the present disclosure is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
Claims
1. An imaging display device, It is characterized in that include: An imaging unit, comprising a plurality of photoelectric conversion elements; a processing unit configured to process a signal sent from the imaging unit; a display unit configured to display an image based on the signal sent from the processing unit; as well as A detection unit is configured to detect environmental information, The imaging unit acquires first image information including the moving object at a first time, The detection unit detects the acceleration information of the moving object acquired by the imaging unit, wherein the processing unit generates first predicted image information of a second time later than the first time and predicting the movement of the moving object based on the first image information and the acceleration information, and The display unit displays an image based on the first predicted image information at the second time as a real-time image of the moving object at the second time.
2. The imaging display device according to claim 1, The imaging unit acquires second image information at a second time, and in, At a third time later than the second time, the imaging unit acquires third image information, and the display unit displays an image based on second predicted image information generated from the second image information and the third image information.
3. The imaging display device according to claim 1, wherein the imaging unit performs an imaging operation for acquiring fourth image information at a fourth time between the first time and the second time, and The first predicted image information is generated at least according to the first image information and the fourth image information.
4. The imaging display device according to claim 1, in, When the number of pieces of image information acquired by the imaging unit per second is defined as the imaging frame rate, and the number of images displayed by the display unit per second is defined as the display frame rate, the imaging frame rate is equal to or greater than the display frame rate.
5. The imaging display device according to claim 1, wherein the processing unit detects the moving object and the stationary object according to the first image information, and identifies the first part detected as the moving object and the second part detected as the stationary object, and in, When generating the first predicted image information, the processing unit performs a resolution reduction process on the second portion.
6. The imaging display device according to claim 1, further comprising a sight line detection unit configured to detect a sight line, in, When generating the first predicted image information, the processing unit performs processing for reducing the resolution of an area other than the line-of-sight area detected by the line-of-sight detection unit.
7. The imaging display device according to claim 6, in, When generating the first predicted image information, the processing unit predicts the visual line area with respect to the second time.
8. The imaging display device according to claim 1, in, When the number of images displayed by the display unit per second is expressed as a display frame rate DFR, a time difference ΔA between image information acquired by the imaging unit at a certain time and predicted image information displayed by the display unit at the certain time satisfies the condition -2 / DFR≤ΔA≤2 / DFR.
9. The imaging display device according to claim 8, wherein the difference ΔA satisfies the condition -4×10 -3 ≤ΔA≤4×10 -3 .
10. The imaging display device of claim 1, wherein the processing unit comprises an artificial intelligence (AI) unit.
11. The imaging display device of claim 10, wherein the AI unit has a deep learning function. 12 . The imaging display device of claim 10 , wherein the AI unit includes a trained model, and the trained model is established based on data acquired from an expert.
13. The imaging display device according to claim 1, wherein each of the plurality of photoelectric conversion elements is capable of detecting light in a visible light region and a near infrared light region, and The processing unit performs processing for converting image information about a near-infrared region of the image information into image information about a visible light region.
14. The imaging display device according to claim 1, wherein the imaging unit includes an imaging area for performing imaging, The display unit includes a display area for performing display, and The centroid of the imaging area and the centroid of the display area coincide with each other.
15. The imaging display device of claim 1, wherein the imaging unit comprises a backside illumination type complementary metal oxide semiconductor (CMOS) image sensor.
16. The imaging display device according to claim 1, in, In the imaging unit, a substrate on which the plurality of photoelectric conversion elements are arranged and a substrate on which a circuit for processing signals from the plurality of photoelectric conversion elements is arranged are stacked.
17. The imaging display device according to claim 1, wherein at least three chips are stacked, the at least three chips including a first chip on which an imaging unit is arranged, a second chip on which a display unit is arranged, and a third chip on which a processing unit is arranged, and The third chip is arranged between the first chip and the second chip.
18. The imaging display device according to any one of claims 1 to 17, wherein the display unit includes an organic light emitting diode (LED) or an inorganic LED as a light emitting element.
19. A wearable device, It is characterized in that include: An imaging display device including an imaging unit having a plurality of photoelectric conversion elements, a processing unit configured to process a signal sent from the imaging unit, a display unit configured to display an image based on the signal sent from the processing unit, and a detection unit configured to detect environmental information; a power supply unit configured to supply power to the imaging display device; as well as interface, configured to wirelessly perform external connection, The imaging unit acquires first image information including a moving object at a first time, the detecting unit detects acceleration information of the moving object acquired by the imaging unit, the processing unit generates first predicted image information about a second time later than the first time, which predicts the movement of the moving object, based on the first image information and the acceleration information, and the display unit displays an image based on the first predicted image information at the second time as a real-time image of the moving object at the second time.
20. An imaging display system, It is characterized in that include: Multiple imaging display devices; as well as a control device configured to control the plurality of imaging display devices, Each of the imaging display devices includes an imaging unit having a plurality of photoelectric conversion elements, a processing unit configured to process a signal sent from the imaging unit, a display unit configured to display an image based on the signal sent from the processing unit, and a detection unit configured to detect environmental information, and The imaging unit acquires first image information including a moving object at a first time, the detecting unit detects acceleration information of the moving object acquired by the imaging unit, the processing unit generates first predicted image information about a second time later than the first time, which predicts the movement of the moving object, based on the first image information and the acceleration information, and the display unit displays an image based on the first predicted image information at the second time as a real-time image of the moving object at the second time.
