Information processing terminal and image processing method

Abstract

To provide an information processing terminal and an image processing method for presenting information about an article to a plurality of recipients in real time without requiring a terminal apparatus for displaying AR (augmented reality) images.SOLUTION: An image display apparatus 1 includes a portable information terminal 2 and a projector 3 for projecting image data output from the portable information terminal 2. The portable information terminal 2 includes cameras 206 and 207, a display 212, and an image processing control unit. The cameras 206 and 207 capture images. The display 212 displays the images captured by the cameras 206 and 207. The image processing control unit comprises a CPU 203, a wireless LAN interface 202, an image processing processor 208, or the like, and causes a communication function unit of an article selected from the image displayed on the display 212 to transmit the image data. The projector 3 projects an information image based on the image data received by the portable information terminal 2.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing terminal and an image processing method, and more particularly to a technique that is effective in promoting understanding of exhibited or displayed items. [Background technology]

[0002] In recent years, services using AR (Augmented Reality) have been proposed. AR takes a two-dimensional image called an AR trigger with a digital camera or a camera built into a smartphone, and displays the image of information linked to the AR trigger in real time by synthesizing it with the image taken by the camera.

[0003] One technology for providing services using AR is to store composite AR images in an image sharing database and provide the AR images via a service providing server, thereby allowing other users other than the user to view the AR images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-229090 A Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, in the technique of Patent Document 1, the AR image cannot be shared in real time among a plurality of users because the composite and displayed AR image is stored in an image sharing database.

[0006] In order for multiple users to share an AR image, each user needs a terminal device that displays the AR image. In other words, the number of terminal devices that display the AR image must be the same as the number of users, and when there are a large number of users, sharing the AR image is difficult.

[0007] An object of the present invention is to provide a technique that can present information about an article to multiple recipients in real time without requiring a terminal device that displays an AR image.

[0008] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0009] Among the inventions disclosed in this application, a brief summary of representative inventions is as follows.

[0010] That is, a typical example is a portable information terminal that receives image data, which is information related to an item, transmitted from outside, and is equipped with a camera that captures an image, a display unit that displays the image captured by the camera, and an image processing control unit that acquires the image data from a selected item and outputs the image data generated from the acquired image data to a projector that projects an information image onto the item or an area that can be viewed together with the item, and the image processing control unit causes the image data to be transmitted to a communication function unit equipped with an item selected from the image displayed on the display unit, and an image obtained by processing or replacing part of the image using the image data is output to the projector as the information image. Effect of the Invention

[0011] The effects obtained by the representative inventions among those disclosed in this application will be briefly described as follows.

[0012] (1) Information about an item can be presented to multiple recipients in real time.

[0013] (2) The above (1) can improve convenience for users. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is an explanatory diagram showing an example of a configuration of an image display device according to a first embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of use of the image display device of FIG. [Diagram 3] 2 is an explanatory diagram showing an example of processing by an image processor included in the image display device of FIG. 1; [Figure 4] 1. FIG. 4 is an explanatory diagram showing an example of a communication sequence with a commodity by the image display device of FIG. [Diagram 5] 4 is an explanatory diagram showing a detection example in flat portion detection in FIG. 3. [Figure 6] 4 is an explanatory diagram showing an example of processing in the projection image data correction of FIG. 3. [Figure 7] 2 is an explanatory diagram showing an example of an image captured by a camera and displayed on a display of the image display device of FIG. 1; [Figure 8] FIG. 2 is an explanatory diagram showing an example of product information displayed by the image display device of FIG. 1; [Figure 9] 1. FIG. 4 is an explanatory diagram showing another example of an image display by the image display device of FIG. [Figure 10] FIG. 10 is an explanatory diagram showing another example of FIG. 9. [Figure 11] 2 is an explanatory diagram showing the relationship between motion detection by the image display device of FIG. 1, a camera shooting state, and a projector projection state. [Figure 12] FIG. 11 is an explanatory diagram showing an example of a configuration of an image display device according to a second embodiment. [Figure 13] FIG. 11 is an explanatory diagram showing an example of the configuration of an image display device according to a third embodiment. [Figure 14]FIG. 13 is an explanatory diagram showing an example of the configuration of an image display system using an image display device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In all the drawings for explaining the embodiments, the same members are generally given the same reference numerals, and repeated explanations thereof will be omitted.

[0016] (Embodiment 1) The embodiment will be described in detail below.

[0017] (Example of configuration of image display device) FIG. 1 is an explanatory diagram showing an example of a configuration of an image display device 1 according to the first embodiment.

[0018] 1, the image display device 1 has a mobile information terminal 2 and a projector 3. The mobile information terminal 2 has a mobile communication interface 201, a wireless LAN interface 202, a CPU (Central Processing Unit) 203, a RAM (Random Access Memory) 204, a flash ROM (Read Only Memory) 205, a camera 206, a camera 207, an image processing processor 208, a sensor 209, a video processor 210, a graphic processor 211, a display 212, a microphone / speaker 213 for voice calls, an external interface 214, an external terminal 215, a beacon transmitting / receiving unit 216, etc. The above-mentioned functional blocks are connected to each other by a common bus 217.

[0019] Projector 3, which is an image display unit, is connected to mobile information terminal 2 via external terminal 215. The projection lens surface, which is the projection port of projector 3, is installed so as to face in approximately the same direction as the lens surfaces of cameras 206 and 207. A mechanism for adjusting the installation may also be employed.

[0020] The presenter holds the image display device 1 in his / her hand and aims the lens faces of the cameras 206 and 207 at the object (hereinafter, referred to as a product) to capture the image. The cameras 206 and 207, which are the camera units, are, for example, stereo cameras. The stereo camera uses the parallax between the two cameras to capture the object and measure the distance between the object and the camera.

[0021] Alternatively, one of the cameras 206 and 207 may be used as a camera for photographing a subject, and the other may be a dedicated camera for measuring the distance to the subject using a method such as TOF (Time Of Flight) or SP (Structured Patter).

[0022] The image processor 208 processes the images and distance data captured by the cameras 206 and 207. The CPU 203, the wireless LAN interface 202, the image processor 208, etc. constitute an image processing control unit.

[0023] The sensor 209 is, for example, a gyro sensor, which detects camera shake during shooting with the cameras 206 and 207, and performs camera shake correction during shooting, enabling stable shooting of the subject.

[0024] The mobile communication interface 201 is responsible for a mobile communication function called 3G (Generation) or 4G, and the wireless LAN interface 202 is responsible for a wireless LAN (Local Area Network) function represented by Wi-fi (registered trademark) etc. The wireless LAN interface 202 may include not only a wireless LAN but also a communication function exemplified by Bluetooth (registered trademark) etc.

[0025] The mobile communication interface 201 and the wireless LAN interface 202 operate complementarily in accordance with the surrounding communication environment. For example, the wireless LAN interface 202 is selected indoors, and the mobile communication interface 201 is selected outdoors.

[0026] The product, which is the subject of the image, also has a communication function unit 300 shown in Fig. 2 that communicates with the image display device 1. In addition to the communication function, this communication function unit 300 has a storage unit (not shown) that stores image data, which is information related to the product, or data such as individual information (described later). Hereinafter, the image display device 1 selects a product that can communicate through the communication function unit 300 provided in the product, communicates with the selected product, and obtains information related to the product.

[0027] The CPU 203 executes programs such as applications and an OS (operating system) stored in a flash ROM 205. When executed, the programs in the flash ROM 205 are expanded in a RAM 204, which is a volatile semiconductor memory.

[0028] The flash ROM 205 is a non-volatile semiconductor memory that allows data to be rewritten, and stores, in addition to programs, product-related information, image data of the product-related information, etc. These data are also expanded in the RAM 204.

[0029] The programs executed by the CPU 203 include control programs for an image processor 208, a video processor 210, and a graphics processor 211 connected to a common bus 217, in addition to applications and the OS.

[0030] The above-mentioned product-related information, that is, image data, is processed by the CPU 203, the video processor 210, and the graphic processor 211, and converted into an information image that can be projected by the projector 3.

[0031] If the product-related information is compressed image data such as MPEG (Moving Picture Experts Group) or JPEG (Joint Photographic Experts Group), the information is decoded by the video processor 210. If the information is hypertext or other data, the graphic processor 211 converts the data into image data. The obtained image data is sent to the projector 3 via the external interface 214 and the external terminal 215.

[0032] The display 212 has, for example, a touch panel, and displays images captured by the cameras 206 and 207 or an application execution screen. It also prompts the user to perform an input operation as necessary, enabling interactive operation on the image display device 1.

[0033] The beacon transmitting / receiving unit 216 transmits a beacon signal to a product and receives a beacon signal transmitted from the product. The beacon signal is used to detect the presence of the product in the vicinity of the image display device 1.

[0034] (Examples of use of image display devices) FIG. 2 is an explanatory diagram showing an example of use of the image display device 1 of FIG.

[0035] 2 shows an example in which the image display device 1 is a smartphone, and the smartphone has a built-in projector 3. Also, in FIG. 2, an example is shown in which a suitcase 6 is used as a product.

[0036] 2, an explainer 4 explaining a suitcase 6, which is a product, holds the image display device 1 and takes a picture with the lens faces of cameras 206, 207 directed toward the suitcase 6. The taken image is displayed on the display 212. Then, on the display screen of the display 212, the explainer 4 selects the product to explain. The product is selected, for example, by touching the image of the suitcase 6 that the explainer is explaining.

[0037] When a suitcase 6, which is a product to be described, is selected, the image display device 1 and the communication function unit 300 of the suitcase 6 communicate with each other using communication signals 21a and 21b. As a result, the image display device 1 obtains information related to the selected suitcase 6 from the suitcase 6.

[0038] Furthermore, the image display device 1 converts image data, which is information related to the suitcase 6 obtained through communication, into an information image and projects it onto the suitcase 6, thereby presenting the projection screen 30, which is an information screen, to the recipients 5a, 5b, and 5c.

[0039] This allows the recipients 5a, 5b, and 5c to receive the explanation from the presenter 4 while simultaneously looking at the suitcase 6 and the projection screen 30. In this way, when multiple recipients view product information, it is not necessary for each of the recipients 5a, 5b, and 5c to have a terminal device such as an AR (Augmented Reality) head mounted display or display device for viewing the product information, so that it is possible to reduce device costs and improve user convenience.

[0040] Example of image processor processing Fig. 3 is an explanatory diagram showing an example of processing by the image processor 208 included in the image display device 1 in Fig. 1. In Fig. 3, an example of processing by the image processor 208 is shown, but the processing shown below may be performed by the image processor 208 and the CPU 203.

[0041] The image processor 208 executes a first process, a second process, and a third process. The first process is a process for performing product recognition and product selection operation reception. The second process is a process for performing distance data reception, flat area detection, and projection area determination. The third process is a process for receiving projection image data, correcting projection image data, and transmitting projection image data.

[0042] In the first process, in product recognition, images captured by the cameras 206 and 207 are processed to recognize products (step S101). For product recognition, the presence of nearby products is detected in advance using beacon communication by the beacon transmission / reception unit 216.

[0043] The beacon transmitting / receiving unit 216 does not transmit toward a specific product, so if there are multiple products that respond to the beacon signal around the image display device 1, the multiple products are detected. Alternatively, image recognition of objects may be performed from images captured by the cameras 206 and 207, and the product may be identified based on the results. Since the presence of products is narrowed down in advance by beacon communication, the calculation load of image recognition is reduced.

[0044] The detected products are presented as selection candidates on the display 212. In this case, the selection candidate products, i.e., the detected products, are displayed in the images captured by the cameras 206, 207 and displayed on the display 212 with a frame or the like as an identification mark, so that the presenter 4 recognizes that they are selection candidates.

[0045] The presenter 4 determines the product to be explained by performing an operation such as touching the product image on the display screen of the image captured by the cameras 206 and 207. In the product determination operation reception, the image processor 208 receives a determination instruction when it determines the product to be explained (step S102).

[0046] In the second process, in distance data reception, distance data is received from the cameras 206 and 207 (step S103). The distance data is, for example, associated with pixel units of the image captured by the cameras 206 and 207. In flat area detection, the distance data is used to determine an area where the deviation of the distance fluctuation is within a preset reference value as a flat area (step S104).

[0047] In the projection area determination, the projection area is determined from the area determined as the flat part (step S105). The projection area may be selected as one flat part, or may be selected as a combination of multiple flat parts.

[0048] In the third process, in receiving the projection image data, image data of information relating to the product obtained by the video processor 210 and the graphic processor 211 is received from the selected suitcase 6, that is, the product determined in the process of step S102 (step S106).

[0049] 2, when the projection screen 30 is projected onto the suitcase 6, which is a commodity, the projection image data is corrected by changing the color tone or converting the shape of the projection image data in consideration of the background color of the projection area, the shape of the projection area, etc. (step S107). This makes the projection screen easier to see.

[0050] In the projection image data transmission, the projection image data corrected in the projection image data correction process is sent to the projector 3 via the external interface 214 in FIG. 1 or the like (step S108).

[0051] 3 shows an example in which the above-mentioned first process, second process, and third process are processed in sequence, but the order of these processes is not limited to this. For example, the first process and the second process, the second process and the third process, the first process and the third process, or the first process to the third process may be processed in parallel.

[0052] <Example of communication between image display device and product> FIG. 4 is an explanatory diagram showing an example of a communication sequence with a product by the image display device 1 of FIG.

[0053] First, the image display device 1 transmits a beacon signal from the beacon transmitting / receiving unit 216 in order to detect a product to be described (step S201). As described above, this beacon signal is not transmitted toward a specific product.

[0054] Therefore, a plurality of products in the vicinity of the image display device 1 that has received the beacon signal return a beacon response indicating that the beacon signal has been received (step S202). This beacon response includes individual information such as a product code or model number that identifies the product.

[0055] Moreover, the beacon response, which is the process in step S202, may use broadcast communication by the wireless LAN interface 202, for example, instead of a beacon signal.

[0056] In the image display device 1, the image processor 208 checks the responses from the surrounding products and determines the product to be explained (step S203). In the process of step S203, the product selected by the presenter 4 in the product decision operation reception (step S102) in FIG. 3 is determined as the product to be explained.

[0057] When the product is determined, the image processing processor 208 acquires individual information of the determined product by image recognition processing. In this case, for example, a table indicating the feature points of each product and the individual information associated with the feature points is stored in the flash ROM 205. The image processing processor 208 acquires the individual information of the determined product based on this table.

[0058] Next, the CPU 203 transmits a data request to the product that matches the acquired individual information (step S204). In this case, the CPU 203 searches for individual information included in the beacon response, and searches for a product that matches the individual information acquired when the product is selected.

[0059] Then, a data request is transmitted to the product whose individual information included in the beacon response matches the acquired individual information. This data request is transmitted, for example, via the wireless LAN interface 202.

[0060] In response to the data request from the image display device 1, the product 6 transmits data of information related to the product (step S205). The image processing device 1 receives the data of information related to the product via the wireless LAN interface 202.

[0061] <Example of flat area detection> FIG. 5 is an explanatory diagram showing a detection example in the flat portion detection of FIG.

[0062] The images captured by cameras 206 and 207 are two-dimensional images, and the corresponding distance data is also two-dimensional data, but in Fig. 5, it is shown one-dimensionally for ease of explanation. In Fig. 5(a) and Fig. 5(b), the horizontal axis indicates the pixel position, and the vertical axis indicates the distance. Moreover, solid line 841a in Fig. 5(a) and solid line 841b in Fig. 5(b) are plots of distance data, respectively.

[0063] In the case of Fig. 5(a), as shown by the solid line 841a, a flat portion in which the deviation in distance falls within δ can be detected in the pixel position range from position X1 to position X2. Also, in the case of Fig. 5(b), as shown by the solid line 841b, in the pixel position range from position X3 to position X4, the distance increases monotonically, but a sloped flat portion in which the deviation excluding the slope falls within δ can be detected. Even the sloped flat portion can be selected as the projection area.

[0064] <Example of Projection Image Data Correction Processing> FIG. 6 is an explanatory diagram showing an example of the process of correcting the projection image data in FIG.

[0065] FIG. 6 shows an example in which three types of correction, color brightness correction, shape correction, and depth correction, are performed as the projection image data correction.

[0066] The color brightness correction performs background color detection and character color correction, the shape correction performs projection area shape detection and image data shape correction, and the depth correction performs projection area depth detection and image data depth correction.

[0067] First, in background color detection, the background color of the projection area of ​​the image captured by the cameras 206 and 207 is detected (step S301). Then, in character color correction, color correction and brightness correction are performed on characters and the like included in the projection image data (step S302). This enhances the contrast with the background and improves the visibility of the projection image.

[0068] Next, in projection area shape detection, the shape of the projection area is detected, for example, when the projection area is a trapezoid, a curved surface, or is configured as separate areas (step S303).

[0069] Thereafter, in image data shape correction, shape correction of the projection image data is performed in accordance with the detected shape of the projection area, and the projection image is placed within the range of the projection area (step S304).

[0070] Then, the depth shape of the projection area is detected by projection area depth detection (step S305). This process is executed, for example, when there is a slope in the distance data.

[0071] Next, image data depth correction is performed on the image based on the detected depth shape (step S306). This correction is, for example, a correction to make the characters larger as they are closer to the back of the projection area. This makes it possible to reduce distortion of the displayed characters.

[0072] <Example of camera image display> FIG. 7 is an explanatory diagram showing an example of an image captured by a camera and displayed on the display 212 of the image display device 1 of FIG.

[0073] Fig. 7(a) shows an example in which the product being explained by the presenter is a suitcase 6, and an image of the exterior of the suitcase 6 is captured. Fig. 7(b) shows an example in which an image of the suitcase 6, which is the product in Fig. 7(a), is captured with the lid open.

[0074] 7(a) and 7(b), a product pointer 62 is displayed on a photographed image of a suitcase 6, which is a product, on the display 212. The presenter 4, for example, touches the product pointer 62 displayed on the display 212 to select the suitcase 6 directly below the product pointer 62.

[0075] As shown in FIG. 7(a) and FIG. 7(b), even the same product may be in different states, such as when the suitcase 6 is closed or when the lid of the suitcase 6 is open.

[0076] In this way, the image display device 1 can select and provide optimal information according to various product conditions.

[0077] <Product information display example> FIG. 8 is an explanatory diagram showing an example of product information displayed by the image display device 1 of FIG.

[0078] Fig. 8(a) shows an example of projecting information onto the images captured by the cameras 206 and 207 in Fig. 7(a). That is, an example is shown in which an information image, that is, an image 31a of information relating to the suitcase 6, is projected onto the exterior of the suitcase 6 in a closed state.

[0079] Fig. 8(b) shows an example of projection of information onto the images captured by the cameras 206 and 207 in Fig. 7(b). That is, in the example shown, when the lid of the suitcase 6 is open, an information image, i.e., an image 31b of information relating to the suitcase 6, is projected onto the inside of the open lid.

[0080] As shown in Figures 8(a) and 8(b), the contents of the images 31a and 31b that provide information are displayed differently depending on the state of the product. In other words, even if the product is the same, the information related to the product is displayed differently depending on the state of the product.

[0081] When the image processor 208 recognizes that the state of the suitcase 6 is as shown in FIG. 8(a), the CPU 203 makes a data request to transmit image data corresponding to the state of FIG. 8(a).

[0082] Furthermore, when the image processor 208 recognizes that the suitcase 6 is in the state shown in FIG. 8(b), the CPU 203 makes a data request to transmit image data corresponding to the state shown in FIG. 8(b).

[0083] Alternatively, image data for all states may be acquired in advance from the suitcase 6 at once, and the CPU 203 may extract image data corresponding to the state of the suitcase 6 and output it to the projector 3. In this case as well, the state of the suitcase 6 is recognized by the image processor 208 from the images captured by the cameras 206, 207.

[0084] This makes it easier for the recipient to understand the features of the product.

[0085] <Another example of product information display 1> FIG. 9 is an explanatory diagram showing another example of image display by the image display device 1 of FIG.

[0086] FIG. 9 shows an example in which the image display device 1 is applied to selling clothes.

[0087] First, a model 64 wearing clothes, which is a product shown on the right side of Fig. 9, is photographed by the image display device 1. Here, the model 64 may be a mannequin, etc. A part of this photographed image is cut out, and a part of the face is replaced with the face 33 of the person being explained, and a projection image 32, which is an information image, is projected by the projector 3 as shown on the left side of Fig. 9.

[0088] The face image 33 of the person being explained to is captured, for example, before capturing an image of the model 64 with a camera. The captured image is stored, for example, in the flash ROM 205 or RAM 204 of the image display device 1. The stored face image 33 is then synthesized with the camera-captured image of the model 64. This has the advantage of making it possible to compare the actual product with a virtual image such as AR.

[0089] <Another example of product information display 2> FIG. 10 is an explanatory diagram showing another example of FIG.

[0090] 10 shows an example in which the image display device 1 is applied to an explanation of a room 66 in a house. Fig. 10(a) shows an example in which light entering through a window 67 provided in the room 66 is displayed as projected images 34a and 34b, which are information images shown by shading.

[0091] For example, when a real estate agent shows a property, a time lapse display is possible by displaying the daily movement of sunlight inside room 66 at high speed as a projected image and presenting it to the person being shown. In this case, data on the movement of sunlight is stored in advance in flash ROM 205 or the like, and based on the data on the movement of sunlight stored in flash ROM 205 or the like, CPU 203 and graphic processor 211 generate a projection image that displays the movement of sunlight at high speed throughout the day, and project it by projector 3.

[0092] 10(b) shows an example in which the wind direction and air volume of an air conditioner 68 installed in a room 66 are projected as information images of arrows 35a, 35b, and 35c. The directions of the arrows 35a, 35b, and 35c indicate the wind direction, and the sizes of the arrows 35a, 35b, and 35c indicate the air volume. For example, the larger the displayed arrow, the greater the air volume.

[0093] In this case, the image display device 1 is applied to, for example, a remote controller that controls an air conditioner 68. In cases where the air conditioner 68 is shared among many recipients, the state of the air from the air conditioner 68 can be easily conveyed, and the air direction, air volume, etc. of the air conditioner 68 can be controlled to a state that satisfies all of the recipients.

[0094] In addition, the control of air conditioners can be applied to consumer use as well as business use. Fig. 10(b) can also be applied to home air conditioners, for example, to a remote controller for home air conditioners.

[0095] Furthermore, the present invention can be applied to consumer products such as refrigerators, where a list of foods stored in the refrigerator can be provided as product-related information, and an image of the information can be projected onto the wall of the refrigerator to be shared among family members.

[0096] In this case, by selecting food from the projected information and linking to an information site on dishes and cooking methods, user convenience can be improved.

[0097] Next, a technique for switching between the cameras 206 and 207 and the projector 3 will be described.

[0098] <Example of synchronous operation of camera and projector> FIG. 11 is an explanatory diagram showing the relationship between the motion detection by the image display device 1 of FIG. 1, the camera shooting state, and the projector projection state.

[0099] 11, motion detection, camera capture, and projector projection are shown from top to bottom, and the horizontal axis indicates time. Motion detection is the result of detection of motion of mobile information terminal 2 by sensor 209, with "Yes" indicating that motion has been detected and "No" indicating that motion has not been detected.

[0100] Camera shooting indicates the shooting state of cameras 206, 207, with "ON" indicating a shooting state, i.e., cameras 206, 207 being activated, and "OFF" indicating a non-shooting state, i.e., cameras 206, 207 being deactivated.

[0101] Projector projection is the projection state of projector 3, with "ON" indicating that projection by projector 3 is occurring, and "OFF" indicating that projection by projector 3 is not occurring.

[0102] The cameras 206, 207 and the projector 3 operate synchronously. The interval between the vertical dotted lines shown in Fig. 11 corresponds to one or several frames of a video image (hereinafter referred to as an image frame), and the cameras 206, 207 and the projector 3 switch their operations ON / OFF with the image frame as a reference unit.

[0103] 11, when the cameras 206 and 207 start taking pictures, the sensor 209 detects the movement from time t0 on the left to between time t1 and time t2. This movement is detected by the sensor 209, which is a gyro sensor as described above.

[0104] 11, is a period in which the presenter 4, for example, moves the portable information terminal 2 to capture a subject product, etc. Then, when the subject product is captured and photographed by the cameras 206, 207, the movement of the portable information terminal 2 stops, and photographing by the cameras 206, 207 is performed.

[0105] At this time, projector projection is not performed, that is, the projector is turned off. In projector projection, strong light is projected from the projector 3, which may interfere with camera photography.

[0106] Therefore, during the camera shooting period, that is, the period from when the sensor 209 detects the movement until the subject product is captured and photographed, the camera shooting can be performed satisfactorily by turning off the projector 3.

[0107] If the motion detection state transitions to no motion between time t1 and time t2, the cameras 206 and 207 perform camera shooting in an image frame period from time t2 to time t3, and store the camera shot images during that period in the mobile information terminal 2. Then, the product recognition (step S101) and product determination operation reception (step S102) of Fig. 3 are performed to prepare image data of information related to the products.

[0108] At time t3, the projector projection is turned on, and a projector projection period in which image data of product information is projected is started. At this time, the camera shooting is turned off. The camera shooting period and the projector projection period are determined by, for example, the CPU 203, which controls the operations of the cameras 206 and 207 and the projector 3.

[0109] As shown at times t4 to t5, in any image frame in which no motion is detected, the camera shooting can be switched ON and the projector projection can be switched OFF. For example, this is the case when the presenter 4 operates to switch products.

[0110] When the motion detection transitions to an enabled state again from time t6, the camera capture is switched ON and the projector projection is switched OFF in conjunction with the transition of the motion detection. In this way, by using the motion detection of the sensor 209 such as a gyro sensor, it is possible to easily automatically detect the period during which the presenter 4 is moving the mobile information terminal 2 to capture the subject product.

[0111] As a result, the presenter can project an image of product-related information onto or near the product to be explained, and present the product-related information to the person being explained in real time. When presenting the product information, the person being explained does not need to have a terminal device or the like, which improves convenience.

[0112] In addition, by projecting an image of information related to a product onto or near the product, multiple people being explained about the product can simultaneously see the product's substance and related information, which promotes understanding of the product's features. Furthermore, when the presenter is explaining another product, he or she can select the target product and change the projection area.

[0113] (Embodiment 2) <Configuration Example of Image Display Device 1> FIG. 12 is an explanatory diagram showing an example of a configuration of an image display device 1 according to the second embodiment.

[0114] In the above-mentioned embodiment 1, the image display device 1 is configured so that the portable information terminal 2 and the projector 3 are integrated as shown in FIG. 1. However, in the present embodiment 1, the portable information terminal 2 and the projector 3 are configured so as to be separated as shown in FIG. 12.

[0115] The mobile information terminal 2 and the projector 3, which are configured separately, are connected to each other via a network. The network is, for example, a wireless network such as a wireless LAN, and the mobile information terminal 2 and the projector 3 are wirelessly connected to each other via an access point 7.

[0116] In this way, by providing the projector 3 separately from the mobile information terminal 2, the degree of freedom in selecting the projector 3 is increased, and it becomes possible to select, for example, a projector with higher brightness.

[0117] As a result, in addition to the effects of the first embodiment, it is possible to provide a projected image with excellent visibility even in a bright environment.

[0118] (Embodiment 3) <Configuration Example of Image Display Device 1> FIG. 13 is an explanatory diagram showing an example of a configuration of an image display device 1 according to the third embodiment.

[0119] 13, the image display device 1 has a mobile information terminal 2, a projector 3, and cameras 8, 8a, and 8b, which are individually separated. The mobile information terminal 2, the projector 3, and the cameras 8, 8a, and 8b are wirelessly connected by a wireless network such as a wireless LAN via an access point 7. The cameras 8, 8a, and 8b are, for example, network cameras. Alternatively, the cameras 8, 8a, and 8b may be PTZ cameras capable of PTZ (Pan Tilt Zoom) control.

[0120] 12 shows an example in which the cameras 8, 8a, and 8b are PTZ cameras, and products and the like are photographed by the cameras 8, 8a, and 8b. The mobile information terminal 2 associates the products and the like with each other by the processing sequence shown in FIG. 3 of the first embodiment.

[0121] Since a PTZ camera or the like is attached to a fixed object such as a building, it is possible to perform stable camera imaging without being affected by camera shake. In addition, a PTZ camera allows for a high degree of freedom in changing the position, angle, and size of a product when photographing it. Furthermore, if the separate projector 3 is installed in a stable location, the projection screen can be stabilized.

[0122] As a result, in addition to the effect of the second embodiment, a more stable image captured by a camera can be obtained.

[0123] (Embodiment 4) Example of an image display system configuration FIG. 14 is an explanatory diagram showing an example of the configuration of an image display system 50 using the image display device 1 according to the fourth embodiment.

[0124] 14, the image display system 50 is composed of, for example, an image display device 1, an access point 7, and a server 9. The image display device 1 is similar to that in the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

[0125] The access point 7 is provided so that the image display device 1 can access the server 9 via a WAN (Wide Area Network) 10. The access point 7 and the image display device 1 are connected to each other via, for example, a wireless LAN.

[0126] The product determination process is the same as that shown in Fig. 3 and Fig. 4 in the first embodiment, but the image display device 1 transmits a data request to the server 9 after the product determination process in step S203 in Fig. 4. The server 9 that receives the data request transmits information related to the requested product to the image display device 1. This enables the image display device 1 to obtain information related to the product to be explained.

[0127] The information related to the product varies depending on the state of the product, i.e., in the case of the suitcase 6 in Fig. 8, whether the lid is open or closed as described above, but if the state of the product is diverse, the amount of information related to the product may increase accordingly. In such a case, the communication load with the product may become too heavy to obtain the information of the product by communicating with the product.

[0128] On the other hand, in the case of the image display system 50 shown in FIG. 14, the product-related information is stored in the server 9, and the information is sent from the server 9 to the image display device 1, so that the above-mentioned problem is eliminated.

[0129] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention.

[0130] For example, in the first to fourth embodiments, a case has been described in which a camera-captured image is displayed on the display 212 of the mobile information terminal 2, and image data of information relating to products, etc. is displayed on the projection screen of the projector 3, but it is also possible to switch between displaying a camera-captured image and image data of information relating to products on the projection screen of the projector 3 over time.

[0131] Furthermore, although an example has been given in which a presenter explains a product to a person being explained about, the purpose of using the image display device 1 is not limited to this, and a similar use is possible if information about any product is to be displayed on or near the product itself.

[0132] In addition, some or all of the functions of the mobile information terminal 2 may be implemented in hardware, for example by designing them in an integrated circuit. Also, they may be implemented in software, in which a microprocessor unit, CPU, etc. interprets and executes an operating program that realizes each function. There is no restriction on the scope of software implementation, and hardware and software may be used together. [Explanation of symbols]

[0133] 1 Image display device 2. Mobile information terminals 3. Projector 7. Access Points 8. Camera 8a Camera 8b Camera 9 Server 50 Image display system 62 Product Pointer 201 Mobile Communication Interface 202 Wireless LAN Interface 203 CPU 204 RAM 205 Flash ROM 206 Camera 207 Camera 208 Image Processor 209 Sensor 210 Video Processor 211 Graphics Processor 212 Display 213 Microphone / Speaker 214 External Interface 215 External terminal 216 Beacon Transmitter / Receiver 217 Common Bus

Claims

1. A portable information terminal that receives image data that is information related to an item transmitted from an external device, A camera for taking images; a display unit that displays an image captured by the camera; an image processing control unit that acquires the image data from the selected item and outputs an information image generated from the acquired image data to a projector that projects the information image onto the item or onto an area where the information image can be viewed together with the item; Equipped with The image processing control unit A portable information terminal that transmits the image data to a communication function unit of an item selected from the image displayed on the display unit, and outputs an image obtained by processing or replacing a portion of the image using the image data to the projector as the information image.

2. 2. The portable information terminal according to claim 1, A portable information terminal, wherein a lens surface of the camera is provided on the same plane as a projection lens surface of the projector.

3. 2. The portable information terminal according to claim 1, The image processing control unit of the portable information terminal sets a flat portion detected from distance data output from the camera as a projection area, and controls the projector to project the information image generated from the image data onto the set projection area.

4. 4. The portable information terminal according to claim 3, The image processing control unit detects a background color of the projection area and corrects the image data so as to enhance contrast between the information image projected onto the projection area and the detected background color.

5. 4. The portable information terminal according to claim 3, The image processing control unit detects a shape of the projection area, and corrects a shape of the information image to be projected in accordance with the detected shape of the projection area.

6. 2. The portable information terminal according to claim 1, an operation control unit for controlling operations of the camera and the projector; The mobile information terminal, wherein the operation control unit stops the projection of the information image by the projector when it is determined that the camera is in a shooting state.

7. 2. The portable information terminal according to claim 1, A communication unit for connecting to a communication network, The image processing control unit requests and acquires the image data relating to the recognized item from a server connected to the communication network via the communication unit.

8. An image processing method using a mobile information terminal including a camera for capturing an image, and an image processing control unit for acquiring image data from a selected item, and outputting an information image generated from the acquired image data to a projector for projecting the information image onto the item or onto an area where the information image can be viewed together with the item, comprising: the camera taking an image; a step in which the image processing control unit causes a communication function unit of an item selected from the image to transmit the image data, and outputs an image obtained by processing or replacing a part of the image using the image data to the projector as the information image; An image processing method comprising the steps of:

9. 9. The image processing method according to claim 8, The step of projecting the information image includes: detecting a flat portion from distance data output from the camera; determining the detected flat portion as a projection area and controlling the projector to project the information image onto the determined projection area; An image processing method comprising the steps of:

10. 9. The image processing method according to claim 8, The step of projecting the information image includes: detecting a flat portion from distance data output from the camera; determining the detected flat portion as a projection area and detecting a background color of the determined projection area; correcting the image data so that a contrast between the information image projected onto the projection area and the detected background color is enhanced; An image processing method comprising the steps of:

11. 9. The image processing method according to claim 8, The step of projecting the information image includes: detecting a flat portion from distance data output from the camera; determining the detected flat portion as a projection area and detecting a shape of the determined projection area; correcting a shape of the information image to be projected in accordance with the detected shape of the projection area; An image processing method comprising the steps of:

Images