Information processing terminal and image processing method

Abstract

To provide an information processing terminal and an image processing method for presenting information about an article to a plurality of recipients in real time without requiring a terminal apparatus for displaying AR (augmented reality) images.SOLUTION: An image display apparatus 1 includes a portable information terminal 2 and a projector 3 for projecting image data output from the portable information terminal 2. The portable information terminal 2 includes cameras 206 and 207, a display 212, and an image processing control unit. The cameras 206 and 207 capture images. The display 212 displays the images captured by the cameras 206 and 207. The image processing control unit comprises a CPU 203, a wireless LAN interface 202, an image processing processor 208, or the like, and causes a communication function unit of an article selected from the image displayed on the display 212 to transmit the image data. The projector 3 projects an information image based on the image data received by the portable information terminal 2.SELECTED DRAWING: Figure 1

Description

Technical Field

[0006] , ,

[0005] , ,

[0001] The present invention relates to an information processing terminal and an image processing method, and particularly relates to a technique effective for promoting the understanding of articles displayed or exhibited.

Background Art

[0002] In recent years, proposals for services using AR (Augmented Reality) have been made. AR captures a two-dimensional image called an AR trigger with a digital camera or a camera built into a smartphone, etc., and synthesizes and displays the image of the information associated with the AR trigger in real time on the camera-captured image.

[0003] As a technique for providing a service using AR, for example, an AR image to be synthesized and displayed is stored in an image sharing database, and the AR image is provided via a service providing server, enabling other users other than the user to view the AR image (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] The object of the present invention is to provide a technology that can present information about an item in real time to multiple subjects without requiring a terminal device for displaying AR images.

[0008] The aforementioned and other objectives and novel features of the present invention will become apparent from the description herein and the accompanying drawings. [Means for solving the problem]

[0009] A brief overview of some of the representative inventions disclosed in this application is as follows:

[0010] In other words, a typical example is a portable information terminal that receives image data, which is information relating to an item transmitted from an external source, and comprises a camera that takes pictures, a display unit that displays the images taken by the camera, and an image processing control unit that acquires the image data from a selected item and outputs an information image generated from the acquired image data to a projector that projects the image onto the item or an area where the item can be seen together with the item, wherein the image processing control unit causes the selected item to transmit the image data to a communication function unit that has been selected from the image displayed on the display unit, and outputs an image obtained by processing or replacing a part of the image using the image data as the information image to the projector. [Effects of the Invention]

[0011] The effects obtained by some of the representative inventions disclosed in this application can be briefly explained as follows.

[0012] (1) Information about the item can be presented to multiple recipients in real time.

[0013] (2) The above (1) can improve user convenience. [Brief explanation of the drawing]

[0014] [Figure 1] This is an explanatory diagram showing an example of the configuration of an image display device according to Embodiment 1. [Figure 2] Figure 1 is an explanatory diagram illustrating an example of how to use the image display device shown. [Figure 3] This is an explanatory diagram showing an example of processing by the image processing processor in the image display device shown in Figure 1. [Figure 4] Figure 1 is an explanatory diagram showing an example of a communication sequence between the image display device and a product. [Figure 5] Figure 3 is an explanatory diagram showing an example of detection in flat area detection. [Figure 6] Figure 3 is an explanatory diagram showing an example of the processing involved in correcting projected image data. [Figure 7] Figure 1 is an explanatory diagram showing an example of a camera-captured image displayed on the image display device. [Figure 8] Figure 1 is an explanatory diagram showing an example of how product information is displayed on the image display device. [Figure 9] This is an explanatory diagram showing another example of image display using the image display device shown in Figure 1. [Figure 10] This is an explanatory diagram showing another example from Figure 9. [Figure 11] This is an explanatory diagram showing the relationship between motion detection by the image display device, camera capture status, and projector projection status in Figure 1. [Figure 12] This is an explanatory diagram showing an example of the configuration of an image display device according to Embodiment 2. [Figure 13] This is an explanatory diagram showing an example of the configuration of an image display device according to Embodiment 3. [Figure 14]It is an explanatory diagram showing an example of the configuration in an image display system using the image display device according to Embodiment 4.

Embodiments for Carrying Out the Invention

[0015] In all the drawings for explaining the embodiments, the same members are in principle denoted by the same reference numerals, and the repeated explanations thereof are omitted.

[0016] (Embodiment 1) Hereinafter, the embodiments will be described in detail.

[0017] 〈Example of Configuration of Image Display Device〉 FIG. 1 is an explanatory diagram showing an example of the configuration in the image display device 1 according to Embodiment 1 of the present invention.

[0018] As shown in FIG. 1, the image display device 1 has a portable information terminal 2 and a projector 3. The portable information terminal 2 includes a mobile communication interface 201, a wireless LAN interface 202, a CPU (Central Processing Unit) 203, a RAM (Random Access Memory) 204, a flash ROM (Read Only Memory) 205, a camera 206, a camera 207, an image processing processor 208, a sensor 209, a video processor 210, a graphic processor 211, a display 212, a microphone / speaker 213 for voice calls, an external interface 214, an external terminal 215, a beacon transceiver 216, etc. Each of the above functional blocks is interconnected by a common bus 217.

[0019] The projector 3, which is an image display unit, is connected to the portable information terminal 2 via the external terminal 215. The projection lens surface, which is the projection port of the projector 3, is installed so as to face approximately the same direction as the lens surfaces of the camera 206 and the camera 207 respectively. Also, a mechanism for adjusting the installation may be adopted.

[0020] The presenter holds the image display device 1 in their hand and points the lens surfaces of cameras 206 and 207 towards the subject object (hereinafter referred to as "product") to take a photograph. Cameras 206 and 207, which are part of the camera unit, are, for example, stereo cameras. Stereo cameras use the parallax between two cameras to photograph the subject and also measure the distance between the subject and the camera.

[0021] Alternatively, one of cameras 206 or 207 may be used to photograph the subject, while the other camera is used as a dedicated camera to measure the distance to the subject using methods such as TOF (Time Of Flight) or SP (Structured Pattern).

[0022] The image processing processor 208 processes the images and distance data captured by cameras 206 and 207. The image processing control unit is composed of the CPU 203, the wireless LAN interface 202, and the image processing processor 208.

[0023] Furthermore, sensor 209 is, for example, a gyro sensor, which detects camera shake during shooting with cameras 206 and 207, performs image stabilization during shooting, and enables stable shooting of subjects.

[0024] The mobile communication interface 201 handles mobile communication functions, such as 3G (Generation) or 4G, while the wireless LAN interface 202 handles wireless LAN (Local Area Network) functions, such as Wi-Fi (registered trademark). The wireless LAN interface 202 may also include communication functions other than wireless LAN, such as Bluetooth (registered trademark).

[0025] The mobile communication interface 201 and the wireless LAN interface 202 operate complementaryly depending on the surrounding communication environment. For example, the wireless LAN interface 202 is selected indoors, and the mobile communication interface 201 is selected outdoors.

[0026] The product being photographed is also equipped with a communication function unit 300, as shown in Figure 2, which communicates with the image display device 1. In addition to the communication function, this communication function unit 300 has a storage unit (not shown) for storing data such as image data or individual product information, which will be described later, that pertains to the product. Hereafter, the image display device 1 selects a product that can communicate through the communication function unit 300 provided in the product, communicates with the selected product, and obtains information related to that product.

[0027] The CPU 203 executes programs such as applications and the OS (operating system) stored in the flash ROM 205. For execution, the programs in the flash ROM 205 are loaded into the RAM 204, which is a volatile semiconductor memory.

[0028] Flash ROM 205 is a non-volatile semiconductor memory that allows data to be rewritten. It stores programs, product-related information, and image data of product-related information. This data is also loaded into RAM 204.

[0029] Furthermore, the programs executed by the CPU 203 include not only applications and the OS, but also control programs for the image processing processor 208, video processor 210, and graphics processor 211, which are connected to the common bus 217.

[0030] The information relating to the aforementioned products, i.e., image data, is processed by the CPU 203, video processor 210, and graphics processor 211, and converted into an information image that can be projected by the projector 3.

[0031] If the product information is compressed image data such as MPEG (Moving Picture Experts Group) or JPEG (Joint Photographic Experts Group), the video processor 210 decodes it. If the data is hypertext or similar, the graphics processor 211 converts it into image data. The resulting image data is sent to the projector 3 via the external interface 214 and external terminal 215.

[0032] The display 212, for example, has a touch panel and displays images captured by cameras 206 and 207, or the application's execution screen. It also prompts the user for input as needed, enabling interactive operation of the image display device 1.

[0033] The beacon transmitting / receiving unit 216 transmits beacon signals to the product and receives beacon signals transmitted from the product. The beacon signals are used to detect the presence of products in the vicinity of the image display device 1.

[0034] <Examples of image display device usage> Figure 2 is an explanatory diagram showing an example of how the image display device 1 in Figure 1 can be used.

[0035] Figure 2 shows an example where the image display device 1 consists of a smartphone, which has a built-in projector 3. Figure 2 also illustrates a suitcase 6 as a product.

[0036] In Figure 2, the presenter 4, who is explaining the product, suitcase 6, holds the image display device 1 and points the lenses of cameras 206 and 207 towards suitcase 6 to take a picture. The captured image is displayed on the display 212. Then, on the display screen of the display 212, the presenter 4 selects the product to be explained. The product is selected, for example, by touching the image of suitcase 6 that the presenter is explaining.

[0037] When the suitcase 6, which is the product to be described, is selected, the image display device 1 and the communication function unit 300 of the suitcase 6 communicate using communication signals 21a and 21b, respectively. As a result, the image display device 1 obtains information related to the selected suitcase 6 from the suitcase 6.

[0038] Furthermore, the image display device 1 converts image data, which is information related to the suitcase 6 obtained from the communication, into an information image and projects it onto the suitcase 6, thereby presenting the projected screen 30, which is an information screen, to the subjects 5a, 5b, and 5c.

[0039] This allows the recipients 5a, 5b, and 5c to receive explanations from the presenter 4 while simultaneously viewing the suitcase 6 and the projected screen 30. In this way, when multiple recipients review product information, it is not necessary to prepare separate terminal devices such as AR (Augmented Reality) head-mounted displays or display devices for each recipient 5a, 5b, and 5c to view the product information. This reduces equipment costs while improving user convenience.

[0040] <Example of processing by an image processing processor> Figure 3 is an explanatory diagram showing an example of processing by the image processing processor 208 of the image display device 1 shown in Figure 1. In Figure 3, an example of processing by the image processing processor 208 is shown, but the processing described below may also be performed by the image processing processor 208 and the CPU 203.

[0041] The image processing processor 208 performs the first, second, and third processes, respectively. The first process is the process of receiving product recognition and product determination operations. The second process is the process of receiving distance data, detecting flat areas, and determining the projection area. The third process is the process of receiving projected image data, correcting the projected image data, and sending out the projected image data.

[0042] In the first process, product recognition involves image processing of images captured by cameras 206 and 207 to recognize the product (step S101). For product recognition, the presence of nearby products is detected in advance using beacon communication by the beacon transceiver unit 216.

[0043] Since the beacon transmitting / receiving unit 216 does not transmit signals to specific products, if there are multiple products around the image display device 1 that respond to beacon signals, multiple products will be detected. Alternatively, image recognition of objects may be performed from the images captured by cameras 206 and 207, and products may be identified based on the results. Because the presence of products is narrowed down in advance by beacon communication, the computational load of image recognition is reduced.

[0044] The detected products are presented as selection candidates on the display 212. In this case, the images captured by cameras 206 and 207 displayed on the display 212 are indicated by surrounding the selection candidate products, i.e., the detected products, with an identification mark such as a frame, so that the presenter 4 recognizes them as selection candidates.

[0045] Presenter 4 performs operations such as touching the product image on the display screen of the images captured by cameras 206 and 207 to determine which product to be presented. Upon receiving the product selection operation, the image processing processor 208 receives a selection instruction indicating that it has determined which product to be presented (step S102).

[0046] In the second process, distance data reception involves receiving distance data from cameras 206 and 207 (step S103). The distance data is, for example, associated with the pixel units of the images captured by cameras 206 and 207. In flat area detection, the distance data is used to determine that an area where the deviation of distance variation is within a preset reference value is a flat area (step S104).

[0047] In determining the projection area, the projection area is determined from the area determined to be a flat area (step S105). The projection area may be a single flat area, or it may be a combination of multiple flat areas.

[0048] In the third process, when receiving projected image data, image data of the product information obtained by the video processor 210 and the graphics processor 211 is received from the selected suitcase 6, i.e., the product determined in the process of step S102 (step S106).

[0049] Projected image data correction, as shown in Figure 2, involves changing the color tone or transforming the shape of the projected image data when projecting the screen 30 onto the suitcase 6, which is the product, taking into consideration the background color and shape of the projection area (step S107). This corrects the projected screen to make it easier to view.

[0050] In the projection image data transmission process, the projection image data corrected by the projection image data correction process is sent to the projector 3 via the external interface 214 in Figure 1 or the like (step S108).

[0051] Note that Figure 3 shows an example where the first, second, and third processes described above are processed in order, but the order of these processes is not limited to this. For example, the first process and the second process, the second process and the third process, the first process and the third process, or the first through third processes may be processed in parallel.

[0052] <Examples of communication between an image display device and a product> Figure 4 is an explanatory diagram showing an example of a communication sequence between the image display device 1 in Figure 1 and a product.

[0053] First, the image display device 1 transmits a beacon signal from the beacon transmitting / receiving unit 216 to detect the product to be described (step S201). As mentioned above, this beacon signal is not transmitted to a specific product.

[0054] Therefore, multiple products located around the image display device 1 that received the beacon signal return a beacon response indicating that they have received the beacon signal (step S202). This beacon response includes individual product information, such as a product code or model number that identifies the product.

[0055] Furthermore, the beacon response process in step S202 may use broadcast communication via, for example, a wireless LAN interface 202 instead of a beacon signal.

[0056] In the image display device 1, the image processing processor 208 checks for responses from surrounding products and determines which product to describe (step S203). In the process of step S203, the product selected by the presenter 4 in the product determination operation reception (step S102) shown in Figure 3 is determined to be the product to describe.

[0057] When a product is determined, the image processing processor 208 acquires the individual product information determined by image recognition processing. In this case, for example, a table showing the characteristic points of each product and the individual product information associated with those characteristic points is stored in a flash ROM 205 or similar. The image processing processor 208 acquires the individual product information of the determined product based on this table.

[0058] Next, the CPU 203 sends a data request for products that match the acquired individual information (step S204). In this case, it searches for individual information included in the beacon response and searches for products that match the individual information acquired when determining the product.

[0059] Then, a data request is sent for products whose individual information included in the beacon response matches the acquired individual information. This data request is sent, for example, via a wireless LAN interface 202.

[0060] Product 6 transmits data related to the product in response to a data request from the image display device 1 (step S205). The image processing device 1 receives data related to the product via the wireless LAN interface 202.

[0061] <Example of detection of flat areas> Figure 5 is an explanatory diagram showing an example of detection in the flat area detection shown in Figure 3.

[0062] The images captured by cameras 206 and 207 are two-dimensional images, and the corresponding distance data is also two-dimensional data. However, in Figure 5, it is shown in one dimension for simplicity of explanation. In Figures 5(a) and 5(b), the horizontal axis represents the pixel position, and the vertical axis represents the magnitude of the distance. The solid lines 841a in Figure 5(a) and 841b in Figure 5(b) are plots of the distance data, respectively.

[0063] In Figure 5(a), as shown by the solid line 841a, a flat area can be detected where the distance deviation of the pixel position is within δ, even in the range from position X1 to position X2. In Figure 5(b), as shown by the solid line 841b, a sloped flat area can be detected where the distance increases monotonically in the range from position X3 to position X4, but the deviation after deducting the slope is within δ. Even in a sloped flat area, it can be selected as a projection region.

[0064] <Example of projection image data correction processing> Figure 6 is an explanatory diagram showing an example of the processing involved in correcting the projected image data in Figure 3.

[0065] Figure 6 shows an example of applying three types of correction to projected image data: color brightness correction, shape correction, and depth correction.

[0066] Color brightness correction performs background color detection and text color correction. Shape correction performs projection area shape detection and image data shape correction. Depth correction performs projection area depth detection and image data depth correction.

[0067] First, in background color detection, the background color of the projection area of ​​the image captured by cameras 206 and 207 is detected (step S301). Then, in character color correction, color correction and brightness correction are performed on characters and other elements included in the projected image data (step S302). This strengthens the contrast with the background and improves the visibility of the projected image.

[0068] Next, in projection region shape detection, the shape of the projection region is detected, such as whether it is a trapezoid, a curved surface, or composed of separate parts (step S303).

[0069] Subsequently, in the image data shape correction step, the shape of the projected image data is corrected to match the shape of the detected projection area, so that the projected image fits within the range of the projection area (step S304).

[0070] Then, the depth shape of the projection area is detected in the projection area depth detection step (step S305). This process is performed, for example, when the distance data has a slope.

[0071] Next, image data depth correction is performed to correct the detected depth shape (step S306). This correction involves, for example, making the characters larger towards the back of the projection area. This reduces distortion of the displayed characters.

[0072] <Example of displaying a camera-captured image> Figure 7 is an explanatory diagram showing an example of a camera-captured image displayed on the display 212 of the image display device 1 shown in Figure 1.

[0073] Figure 7(a) shows an example where the product being described by the presenter is a suitcase 6, and the exterior of the suitcase 6 is photographed. Figure 7(b) shows an example where the lid of the suitcase 6, the product in Figure 7(a), is open when photographed.

[0074] In Figures 7(a) and 7(b), the display 212 shows a product pointer 62 on a photograph of the product, a suitcase 6. The presenter 4 selects the suitcase 6 directly below the product pointer 62 by, for example, touching the product pointer 62 displayed on the display 212.

[0075] As shown in Figures 7(a) and 7(b), even the same product may be in different states, such as when the suitcase 6 is closed or when the lid of the suitcase 6 is open.

[0076] In this way, the image display device 1 can select and provide the most appropriate information according to the various conditions of the product.

[0077] <Example of product information display> Figure 8 is an explanatory diagram showing an example of how product information is displayed by the image display device 1 shown in Figure 1.

[0078] Figure 8(a) shows an example of information projection onto the images captured by cameras 206 and 207 in Figure 7(a). Specifically, it shows an example where an information image, i.e., an image 31a of information related to the suitcase 6, is projected onto the exterior of the suitcase 6 in a closed state.

[0079] Figure 8(b) shows an example of information projection onto the images captured by cameras 206 and 207 in Figure 7(b). Specifically, it shows an example where, with the lid of the suitcase 6 open, an information image, i.e., an image 31b of information related to the suitcase 6, is projected onto the inside of the open lid.

[0080] As shown in Figures 8(a) and 8(b), the content of images 31a and 31b, which represent information, is displayed differently depending on the condition of the product. In other words, even for the same product, the content of the information related to the product is displayed differently depending on its condition.

[0081] When the image processing processor 208 recognizes that the state of suitcase 6 is as shown in Figure 8(a), the CPU 203 requests data to send image data corresponding to the state shown in Figure 8(a).

[0082] Furthermore, when the image processing processor 208 recognizes that suitcase 6 is as shown in Figure 8(b), the CPU 203 requests data to send image data corresponding to the state shown in Figure 8(b).

[0083] Alternatively, image data of all states may be acquired from the suitcase 6 in advance, and the CPU 203 may extract image data corresponding to the state of the suitcase 6 and output it to the projector 3. In this case as well, the state of the suitcase 6 is recognized by the image processing processor 208 based on images captured by cameras 206 and 207.

[0084] This makes it easier for the person being explained to understand the product's features.

[0085] <Another example of how to display product information 1> Figure 9 is an explanatory diagram showing another example of image display by the image display device 1 shown in Figure 1.

[0086] Figure 9 shows an example of applying the image display device 1 to the sale of clothing.

[0087] First, a model 64 wearing the clothing shown on the right side of Figure 9 is photographed by the image display device 1. Here, the model 64 may be a mannequin or the like. A portion of this photographed image is cropped, and the face portion of that cropped image is replaced with the face 33 of the person being explained to. Then, as shown on the left side of Figure 9, the projector 3 projects the information image, the projected image 32.

[0088] The facial image 33 of the person being described is captured, for example, before the camera captures the model 64. The captured image is stored, for example, in the flash ROM 205 or RAM 204 of the image display device 1. Then, the stored facial image 33 is combined with the camera image of the model 64. This has the advantage of allowing comparison between the actual product and a virtual image such as AR.

[0089] <Another example of displaying product information 2> Figure 10 is an explanatory diagram showing another example from Figure 9.

[0090] Figure 10 shows an example of applying the image display device 1 to a description of a room 66 in a house. Figure 10(a) shows an example of displaying the light entering the room 66 through a window 67 as projected images 34a and 34b, which are information images that show the light with a shaded pattern.

[0091] For example, in a property tour conducted by a real estate agent, a time-lapse display can be achieved by rapidly displaying the movement of sunlight throughout the day inside room 66 as a projected image and presenting it to the person being shown. In this case, the data on the movement of sunlight is stored in advance in a flash ROM 205, and based on the data on the movement of sunlight stored in the flash ROM 205, the CPU 203 and graphics processor 211 generate a projection image that rapidly displays the movement of sunlight throughout the day, which is then projected by the projector 3.

[0092] Figure 10(b) shows an example of projecting the airflow direction and volume of an air conditioner 68 installed in room 66 as informational images represented by arrows 35a, 35b, and 35c. The direction of arrows 35a, 35b, and 35c indicates the airflow direction, and the size of arrows 35a, 35b, and 35c indicates the airflow volume. For example, the larger the displayed arrow, the greater the airflow volume.

[0093] In this case, the image display device 1 is applied, for example, to a remote controller that controls an air conditioner 68. When the air conditioner 68 is shared among many people receiving explanations, the image display device 1 can clearly convey the airflow characteristics of the air conditioner 68, and the airflow direction and volume of the air conditioner 68 can be controlled to a state that satisfies all of the people receiving explanations.

[0094] Furthermore, the application of this air conditioner control technology is applicable not only to commercial use but also to consumer use. Figure 10(b) is applicable even to household air conditioners; for example, it can be applied to the remote controller of a household air conditioner.

[0095] Furthermore, it can be applied to consumer use, such as refrigerators. A list of food items stored in the refrigerator is provided as product information, and an image of this information is projected onto the refrigerator wall for sharing among family members.

[0096] In this case, selecting food items from the projected information and linking to websites with information on recipes and cooking methods can improve user convenience.

[0097] Next, we will explain the switching technology for cameras 206, 207, and projector 3.

[0098] <Example of synchronized operation of camera and projector> Figure 11 is an explanatory diagram showing the relationship between motion detection by the image display device 1 in Figure 1, the camera capture state, and the projector projection state.

[0099] In Figure 11, motion detection, camera capture, and projector projection are shown from top to bottom, respectively, and the horizontal axis represents time. Motion detection is the result of motion detection of the mobile information terminal 2 by sensor 209, where "Yes" indicates motion was detected and "No" indicates motion was not detected.

[0100] The camera status indicates the shooting status of cameras 206 and 207. "ON" indicates that shooting is in progress, meaning cameras 206 and 207 are activated, while "OFF" indicates that shooting is not in progress, meaning cameras 206 and 207 are not activated.

[0101] Projector projection refers to the projection status of Projector 3. "ON" indicates that Projector 3 is projecting, and "OFF" indicates that Projector 3 is not projecting.

[0102] Cameras 206, 207 and projector 3 operate in synchronous mode. The intervals between the vertical dotted lines shown in Figure 11 represent one or more frames of the projected image (hereinafter referred to as image frames), and cameras 206, 207 and projector 3 can be switched ON / OFF based on the image frame as the reference unit.

[0103] In Figure 11, with camera capture by cameras 206 and 207 in progress, sensor 209 detects movement from time t0 on the left through to between time t1 and time t2. As mentioned earlier, this movement is detected by sensor 209, which is a gyro sensor, as it detects the movement of the mobile information terminal 2.

[0104] When sensor 209 detects movement, that is, when motion is detected as shown in Figure 11, it is the period when, for example, the presenter 4 is moving the personal information terminal 2 to capture the subject, such as a product. When the subject is captured and the cameras 206 and 207 take a picture, the movement of the personal information terminal 2 stops, and the cameras 206 and 207 take the picture.

[0105] In this state, projector projection is not performed, i.e., it is turned OFF. When projector projection is enabled, strong light is projected from projector 3, which can interfere with camera photography.

[0106] Therefore, during the camera shooting period, that is, the period from when the sensor 209 detects movement until the subject product is captured and photographed, turning off the projector 3 allows for good camera shooting.

[0107] If the motion detection state changes to "none" between time t1 and time t2, cameras 206 and 207 perform camera capture during the image frame period from time t2 to time t3 and save the camera images taken during that period to the portable information terminal 2. Then, product recognition (step S101) and product decision operation reception (step S102) shown in Figure 3 are performed to prepare image data of product-related information.

[0108] When time t3 is reached, the projector projection is turned ON, and the projector projection period begins during which image data related to the product is projected. At this time, camera shooting is OFF. The camera shooting period and the projector projection period are determined by, for example, the CPU 203, which controls the operation of cameras 206, 207 and projector 3.

[0109] As shown in time t4~t5, it is possible to switch camera shooting ON and projector projection OFF at any image frame where motion detection is not enabled. For example, this can be done when the product is switched by the operator 4.

[0110] When motion detection is activated again at time t6, the camera is switched ON and the projector projection is switched OFF in conjunction with the transition of motion detection. In this way, by using motion detection from sensors 209 such as the gyro sensor, it is possible to easily automatically detect the period during which the presenter 4 is moving the mobile information terminal 2 to capture the subject product.

[0111] As a result, the presenter can project images of product-related information onto or near the product being explained, allowing the information to be presented to the person being explained to in real time. When presenting product information, the person being explained to does not need to have a terminal device, thus improving convenience.

[0112] Furthermore, by projecting images of product-related information onto or near the product, multiple people receiving explanations can simultaneously view information related to the actual product, thereby facilitating their understanding of the product's features. In addition, when explaining a different product, the presenter can select the target product and change the projection area.

[0113] (Embodiment 2) <Example configuration of image display device 1> Figure 12 is an explanatory diagram showing an example of the configuration of the image display device 1 according to this second embodiment.

[0114] In the first embodiment described above, as shown in Figure 1, the image display device 1 was configured as an integrated unit consisting of a mobile information terminal 2 and a projector 3. However, in this first embodiment, as shown in Figure 12, the mobile information terminal 2 and the projector 3 are configured as separate units.

[0115] The separate portable information terminal 2 and projector 3 are connected to a network, for example. The network is a wireless network such as a Wi-Fi network, and they are wirelessly connected via an access point 7.

[0116] In this way, by separating the projector 3 from the portable information terminal 2, the degree of freedom in selecting the projector 3 is increased, making it possible to select, for example, a projector with higher brightness.

[0117] As a result, in addition to the effects of Embodiment 1 described above, it is possible to provide a projected image with excellent visibility even in bright environments.

[0118] (Embodiment 3) <Example configuration of image display device 1> Figure 13 is an explanatory diagram showing an example of the configuration of the image display device 1 according to this third embodiment.

[0119] As shown in Figure 13, the image display device 1 includes a portable information terminal 2, a projector 3, and cameras 8, 8a, and 8b, which are configured as individually separate components. The portable information terminal 2, projector 3, and cameras 8, 8a, and 8b are wirelessly connected via a wireless network such as a wireless LAN through, for example, an access point 7. The cameras 8, 8a, and 8b are, for example, network cameras. Alternatively, the cameras 8, 8a, and 8b may be PTZ (Pan Tilt Zoom) cameras capable of PTZ control.

[0120] Figure 12 shows an example where cameras 8, 8a, and 8b are PTZ cameras, and products and other items are photographed by these cameras. The portable information terminal 2 associates the products and other items with the images using the processing sequence shown in Figure 3 of Embodiment 1.

[0121] PTZ cameras, being mounted on fixed structures such as buildings, allow for stable camera imaging without being affected by camera shake. Furthermore, PTZ cameras offer a high degree of flexibility in adjusting the position, angle, and size of the product during camera photography. Additionally, by placing the separate projector 3 in a stable location, the projection image can be stabilized.

[0122] As a result, in addition to the effects of the second embodiment described above, more stable camera images can be obtained.

[0123] (Embodiment 4) <Example of an image display system configuration> Figure 14 is an explanatory diagram showing an example of the configuration of an image display system 50 using the image display device 1 according to this embodiment 4.

[0124] As shown in Figure 14, the image display system 50 is composed of, for example, an image display device 1, an access point 7, and a server 9. The image display device 1 is the same as that shown in Figure 1 of Embodiment 1, so its description is omitted.

[0125] Access point 7 is provided to allow the image display device 1 to access server 9 via WAN (Wide Area Network) 10. Access point 7 and image display device 1 are connected, for example, by wireless LAN.

[0126] The process for determining the product is the same as shown in Figures 3 and 4 of Embodiment 1, but the image display device 1 sends a data request to the server 9 after the product determination process in step S203 of Figure 4. Upon receiving the data request, the server 9 sends information relating to the requested product to the image display device 1. As a result, the image display device 1 can obtain information relating to the product to be described, etc.

[0127] Information about a product varies depending on its condition, such as whether the lid is open or closed in the case of suitcase 6 in Figure 8, as mentioned above. However, if there are many different conditions for a product, the amount of information related to that product can increase accordingly. In such cases, obtaining information about the product through communication with the product may result in an excessively heavy communication load.

[0128] On the other hand, in the case of the image display system 50 shown in Figure 14, information related to the product is stored in the server 9, and the server 9 sends the information to the image display device 1, so the above-mentioned problem is resolved.

[0129] Although the present invention has been specifically described above based on embodiments, it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from its essence.

[0130] For example, in embodiments 1 to 4, we described a case where the camera image is displayed on the display 212 of the mobile information terminal 2, and image data of product information is displayed on the projection screen of the projector 3. However, it is also possible to display the camera image and the image data of product information on the projection screen of the projector 3 in a time-series switching manner.

[0131] Furthermore, while an example was given of a presenter explaining a product to a person being explained to, the purpose of using the image display device 1 is not limited to this. Similar uses are possible when displaying information about any product on or near the device itself.

[0132] Furthermore, some or all of the functions of the mobile information terminal 2 may be implemented in hardware, for example, by designing them using an integrated circuit. Alternatively, the functions may be implemented in software, where a microprocessor unit, CPU, etc., interprets and executes the operating programs that realize each function. The scope of software implementation is not limited, and hardware and software may be used in combination. [Explanation of symbols]

[0133] 1. Image display device 2. Mobile information terminals 3 Projector 7 Access Points 8 cameras 8a camera 8b Camera 9 Servers 50 Image Display Systems 62 Product Pointers 201 Mobile Communication Interface 202 Wireless LAN Interface 203 CPU 204 RAM 205 Flash ROM 206 Camera 207 Camera 208 Image Processing Processors 209 Sensors 210 Video Processors 211 Graphics Processors 212 displays 213 Microphone / Speaker 214 External Interface 215 External terminals 216 Beacon Transmitter / Receiver 217 Common Bus

Claims

1. A portable information terminal that receives image data, which is information about an item transmitted from an external source, A camera for taking pictures, A display unit that displays the image captured by the aforementioned camera, An image processing control unit that acquires the image data from the selected item and outputs the information image generated from the acquired image data to a projector that projects the image onto the item or an area where it can be seen together with the item. Equipped with, The image processing control unit, A portable information terminal that outputs an information image generated from the image data obtained by processing or replacing a portion of the captured image taken by the aforementioned camera to the projector.

2. In the portable information terminal according to claim 1, A portable information terminal wherein the lens surface of the camera is located on the same plane as the projection lens surface of the projector.

3. In the portable information terminal according to claim 1, A portable information terminal comprising: an image processing control unit which sets a flat area detected from distance data output from the camera as a projection area, and controls the projector to project the information image generated from the image data onto the set projection area.

4. In the portable information terminal according to claim 3, The image processing control unit detects the background color of the projection area and corrects the image data so that the contrast between the information image projected onto the projection area and the detected background color is increased, in a portable information terminal.

5. In the portable information terminal according to claim 3, The image processing control unit detects the shape of the projection area and corrects the shape of the projected information image to match the detected shape of the projection area, in a portable information terminal.

6. In the portable information terminal according to claim 1, It has an operation control unit that controls the operation of the camera and the projector, The aforementioned operation control unit stops the projection of the information image by the projector when it determines that the camera is in a shooting state.

7. In the portable information terminal according to claim 1, It has a communication unit that connects to a communication network, The image processing control unit requests and obtains the image data relating to the recognized item from a server connected to the communication network via the communication unit, and is a portable information terminal.

8. An image processing method using a portable information terminal comprising: a camera for taking images; and an image processing control unit for acquiring image data from a selected item and outputting an information image generated from the acquired image data to a projector that projects the image onto the item or an area where it can be seen together with the item; The camera takes the step of capturing an image, The image processing control unit outputs the generated information image to the projector from the image data obtained by processing or replacing a portion of the captured image taken by the camera, An image processing method having the following characteristics.

9. In the image processing method according to claim 8, The step of projecting the aforementioned information image is: The steps include detecting a flat area from the distance data output from the camera, The steps include determining the detected flat area as the projection area and controlling the projector to project the information image onto the determined projection area, An image processing method having the following characteristics.

10. In the image processing method according to claim 8, The step of projecting the aforementioned information image is: The steps include detecting a flat area from the distance data output from the camera, The steps include determining the detected flat area as the projection region and detecting the background color of the determined projection region, The steps include correcting the image data so that the contrast between the information image projected onto the projection area and the detected background color is increased, An image processing method having the following characteristics.

11. In the image processing method according to claim 8, The step of projecting the aforementioned information image is: The steps include detecting a flat area from the distance data output from the camera, The steps include determining the detected flat area as the projection region and detecting the shape of the determined projection region, A step of correcting the shape of the projected information image to match the shape of the detected projection area, An image processing method having the following characteristics.

12. In the portable information terminal described in claim 1, The aforementioned mobile information terminal further includes a memory for storing the user's facial image, A portable information terminal in which an image obtained by processing or replacing a part of the aforementioned captured image is an image in which a part of the aforementioned captured image has been replaced with the face image stored in the memory.

Images