Display method and display system

Abstract

To correctly detect touch by a pointer on a display surface and avoid displaying an extra line.SOLUTION: A projector 10 includes a first acquisition section 110a, a second acquisition section 110b, a first determination section 110c, and a display control section 110e. The first acquisition section 110a sequentially acquires a first captured image and a second captured image on a display surface. The second acquisition section 110b acquires a first brightness value and a first position of a bright point contained in the first captured image and a second brightness value and a second position of a bright point contained in the second captured image. The first determination section 110c determines whether the second brightness value is larger than the first brightness value by a first threshold or more and the second position is closer to a camera than the first position. The display control section 110e displays, on the display surface, a display image containing a drawn image showing a drawn line by a pointer when a determination result is negative, and displays a display image not containing the drawn image when the determination result is positive.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to a display method and a display system.

Background Art

[0002] When an interactive projector touches an indicator such as a finger on a display surface where an image is to be displayed and draws a line forming a character or a figure, it displays an image of the drawn line corresponding to the operation. The touch on the display surface means a state where the distance between the tip of the indicator and the display surface is less than or equal to a predetermined threshold value, and is not limited to a state where the tip of the indicator actually touches the display surface. As an example of an interactive projector, a touch is detected by imaging, with a camera, a bright spot generated when an indicator crosses light such as IR (InfraRed) light irradiated along the display surface in order to detect a touch by the indicator. The light irradiated along the display surface is called a light curtain, and an irradiation device that irradiates the light curtain is called a light curtain unit or a finger touch unit. Patent Document 1 discloses a technique for detecting a touch on a display surface such as a screen when the change in the peak value of the amount of reflected light stops after detecting the reflected light from an operating object such as an indicator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique disclosed in Patent Document 1, there is a case where the peak value fluctuates even when the indicator touches the display surface due to a change in the reflected light from the display surface or a change in the way the light hits the indicator, and there is a risk that the touch of the indicator on the display surface cannot be correctly detected. [Means for solving the problem]

[0005] One aspect of the display method of the present disclosure is to acquire a first image captured by a camera capturing an area including a display surface; to acquire a second image captured by the camera capturing an area including the display surface after acquiring the first image; to acquire a first brightness value indicating the brightness of a first bright spot included in the first image captured by reflected light from an indicator for drawing lines on the display surface, and a first position indicating the coordinates of the first bright spot in the first image captured; and a second brightness value indicating the brightness of a second bright spot included in the second image captured by reflected light from the indicator. The method includes obtaining a second position indicating the coordinates of the second bright spot in a second captured image, and if the first condition is met, such that the second brightness value is greater than or equal to a first threshold and the second position is closer to the camera than the first position, displaying a first display image on the display surface that does not include a drawing image representing the line by the indicator at a third position corresponding to the second position in the image displayed on the display surface, and if the first condition is not met, displaying a second display image on the display surface that includes the drawing image at the third position.

[0006] Furthermore, one aspect of the display system of the present disclosure includes a camera that images an area including a display surface, acquiring a first image captured by the camera of the area, acquiring a second image captured by the camera of the area after acquiring the first image, acquiring a first luminance value indicating the luminance of a first bright spot included in the first image captured by reflected light from an indicator for drawing lines on the display surface, and a first position indicating the coordinates of the first bright spot in the first image captured, and a second luminance value indicating the luminance of a second bright spot included in the second image captured by reflected light from the indicator, and A display device that performs the following: obtaining a second position indicating the coordinates of the second bright spot in a second captured image; if the second brightness value is greater than or equal to a first threshold and the second position is closer to the camera than the first position, displaying a first display image on the display surface that does not include a drawing image representing the line by the indicator at a third position corresponding to the second position in the image to be displayed on the display surface; and if the first condition is not met, displaying a second display image including the drawing image at the third position on the display surface. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example configuration of the display system 1 according to an embodiment of the present disclosure. [Figure 2] This diagram illustrates the IR light emitted from the irradiating device 20A and the irradiating device 20B, respectively, in order to detect the position of the indicator 30 in the display system 1. [Figure 3] This diagram illustrates the IR light emitted from the irradiating device 20A and the irradiating device 20B, respectively, in order to detect the position of the indicator 30 in the display system 1. [Figure 4] This figure shows an example of a line segment L1 drawn using the indicator 30. [Figure 5] This diagram illustrates the change in the positional relationship between the indicator 30 and the display surface SS during the process of drawing line segment L1. [Figure 6] This diagram illustrates the problems with conventional interactive projectors. [Figure 7] This diagram shows an example configuration of the projector 10 included in the display system 1. [Figure 8] This diagram illustrates the positional relationship between bright spot PA, bright spot PB, and reference position P0. [Figure 9] This is a flowchart showing the flow of the display method executed by the processing unit 110 according to the program PRA. [Modes for carrying out the invention]

[0008] The embodiments described below are subject to various technically preferred limitations. However, the embodiments of this disclosure are not limited to those described below. 1. Embodiment Figure 1 shows an example configuration of a display system 1 according to one embodiment of the present disclosure. The display system 1 includes projectors 10A and 10B, and illumination devices 20A and 20B. An image supply device (not shown) is connected to projectors 10A and 10B by wire or wireless. Projector 10A displays a display image PGA corresponding to image data shared from the image supply device (not shown) on a display surface SS such as a projection screen. Similarly, projector 10B displays a display image PGB corresponding to image data shared from the image supply device (not shown) on the display surface SS. Projectors 10A and 10B are installed on the ceiling of the room where the display surface SS is installed. Examples of image data supplied from the image supply device to each of projectors 10A and 10B include image data representing materials for meetings, etc., or image data representing teaching materials for schools, etc. A specific example of the image supply device is, for example, a personal computer.

[0009] When a user brings an indicator 30 close to the display surface SS of projector 10A to draw characters or figures, projector 10A displays a display image PGA on the display surface SS, which includes a drawing image showing the user's lines. The display image PGA is an image in which a drawing image showing the user's lines is superimposed on an image represented by image data shared from an image supply device. In this embodiment, the indicator 30 is the user's finger, but it may also be a stylus. Similarly, when a user brings an indicator 30 close to the display surface SS of projector 10B to draw characters or figures, projector 10B displays a display image PGB on the display surface SS, which includes a drawing image showing the user's lines. The display image PGB is also an image in which a drawing image showing the user's lines is superimposed on an image represented by image data shared from an image supply device. In other words, projector 10A and projector 10B are interactive projectors. Hereafter, when it is not necessary to distinguish between projector 10A and projector 10B, projector 10A and projector 10B will be referred to as projector 10. In the following, as shown in Figure 1, the coordinate axis along the normal of the display surface SS will be referred to as the Z-axis, the coordinate axis along the vertical direction will be referred to as the Y-axis, and the coordinate axis perpendicular to the Y-axis and Z-axis will be referred to as the X-axis.

[0010] Illumination device 20A is a finger touch unit paired with projector 10A. Illumination device 20B is a finger touch unit paired with projector 10B. As shown in Figure 1, illumination devices 20A and 20B are arranged above the display surface SS so as to be aligned in the X-axis direction. Illumination devices 20A and 20B are equipped with IR light sources that emit IR light. Each of illumination devices 20A and 20B irradiates the display surface SS with IR light that forms a light curtain. When a user brings the indicator 30 close to the display surface SS to draw characters or figures, and the indicator 30 crosses the light curtain, reflected light from the light curtain by the indicator 30 is generated. When an image of a predetermined range including the display surface SS is captured by a camera in this state, a bright spot corresponding to the reflected light from the light curtain by the indicator 30 is captured in the image. The display system 1 detects the line drawn by the user using the indicator 30 by tracking the trajectory of this bright spot.

[0011] Figures 2 and 3 illustrate the IR light emitted from the irradiating devices 20A and 20B, respectively. Figure 2 shows the IR light emitted from the irradiating devices 20A and 20B as viewed from the Z-axis direction. Figure 3 shows the IR light emitted from the irradiating devices 20A and 20B as viewed from the Y-axis direction. As shown in Figures 2 and 3, the irradiating device 20A emits IR light LA ​​in a fan shape along the display surface SS, at a distance HA in the Z-axis direction from the display surface SS. IR light LA ​​is an example of the first light in this disclosure. Similarly, the irradiating device 20B emits IR light LB in a fan shape along the display surface SS, at a distance HB in the Y-axis direction from the display surface SS. IR light LB is an example of the second light in this disclosure.

[0012] As shown in Figures 2 and 3, the light curtain formed by IR light LA ​​and the light curtain formed by IR light LB partially overlap when viewed from the Z-axis direction. It is preferable that there is no gap in the Z-axis direction between the light curtain formed by IR light LA ​​and the light curtain formed by IR light LB, that is, that distance HA and distance HB coincide. However, distance HA and distance HB may not coincide due to differences in the initial settings of the irradiation devices 20A and 20B. In addition, curvature is likely to occur at the ends of the fan-shaped light curtain, and distance HA and distance HB may not coincide due to this curvature.

[0013] Assuming that distances HA and HB do not coincide, a line segment L1 is drawn on the display surface SS using the indicator 30, from the starting point PS to the ending point PE, as shown in Figure 4. Figure 5 is a diagram illustrating the change in the positional relationship between the indicator 30 and the display surface SS during the process of drawing the line segment L1. State ST1 in Figure 5 is a state where the indicator 30 is at a sufficient distance from the display surface SS, and no bright spot corresponding to the indicator 30 is captured in the image of the display surface SS in state ST1.

[0014] When the user brings the indicator 30 closer to the display surface SS to draw line segment L1, the indicator 30 crosses the IR light LB, as shown in state ST2 in Figure 5. If the indicator 30 is brought even closer to the display surface SS from state ST2, the indicator 30 crosses both the IR light LB and the IR light LA, as shown in state ST3 in Figure 5. In state ST2 or state ST3, the captured image of the display surface SS shows bright spots corresponding to the reflected light from the indicator 30. The user draws line segment L1 by moving the indicator 30 along the display surface SS from the starting point PS to the ending point PE while maintaining the state where the indicator 30 crosses both the IR light LB and the IR light LA, i.e., state ST3. When the indicator 30 reaches the ending point PE, the user moves the indicator 30 away from the display surface SS, and the relationship between the indicator 30 and the display surface SS changes from state ST3 to state ST2, and then from state ST2 to state ST1.

[0015] The process of changing from state ST2 to state ST3 corresponds to the start of writing of line segment L1, and the process of changing from state ST3 to state ST2 corresponds to the end of writing of line segment L1. In the process of changing from state ST2 to state ST3 and the process of changing from state ST3 to state ST2, the amount of light of the reflected light by the indicator 30 changes greatly, and the position of the bright spot corresponding to the reflected light in the captured image also changes. In a conventional interactive projector, the change in the amount of light of this reflected light and the change in the position of the bright spot are regarded as the movement of the bright spot, and as shown in FIG. 6, there may be cases where an extra line segment L2 connected to the starting point PS and an extra line segment L3 connected to the ending point PE are drawn. The projector 10 in the present embodiment is configured not to draw an extra line at the start and end of writing a line drawn using the indicator 30. The projector 10 is an example of the display device in the present disclosure.

[0016] FIG. 7 is a diagram showing a configuration example of the projector 10. As shown in FIG. 7, the projector 10 includes a processing device 110, a projection device 120, a camera 130, and a storage device 140. In addition to the processing device 110, the projection device 120, the camera 130, and the storage device 140, the projector 10 includes a communication circuit for communicating with an image supply device by wire or wirelessly, and an input device such as a keyboard for receiving various input operations of the user. However, since the communication device and the input device are less relevant to the present disclosure, they are not shown in FIG. 7.

[0017] The processing device 110 includes a processor such as a CPU (Central Processing Unit), that is, a computer. The processing device 110 may be composed of a single processor or a plurality of processors. The processing device 110 functions as a control center of the projector 10 by operating according to the program PRA stored in the storage device 140.

[0018] The projection device 120 includes a projection lens, a liquid crystal driving unit, a liquid crystal panel, and a light source unit. In FIG. 7, the illustration of the projection lens, the liquid crystal driving unit, the liquid crystal panel, and the light source unit is omitted. The liquid crystal driving unit drives the liquid crystal panel according to the image data supplied from the processing device 110, thereby drawing the display image represented by this image data on the liquid crystal panel. The light source unit includes a light source such as, for example, a halogen lamp or a laser diode. The light from the light source unit is modulated for each pixel in the liquid crystal panel and projected as image light through the projection lens. When this image light is projected onto the display surface SS, the display image is displayed on the display surface SS. Note that the projection device 120 may have any configuration as long as it can project image light onto the display surface SS, and may be configured to include a digital micromirror device (DMD) instead of the liquid crystal driving unit and the liquid crystal panel. The liquid crystal panel and the DMD are an example of the display panel in the present disclosure.

[0019] The camera 130 is, for example, an IR camera, and its orientation has been adjusted so that the optical axis faces the center of the display surface SS. The camera 130 includes, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) which is an image pickup device that converts the light collected by an optical system such as a lens into an electrical signal. An IR light filter for receiving IR light is attached to the camera 130. Under the control of the processing device 110, the camera 130 images an image of a predetermined range including the display surface SS at a predetermined period. Each time the camera 130 images an image of a predetermined range including the display surface SS, the camera 130 outputs the image data representing the captured image to the processing device 110.

[0020] The storage device 140 is a recording medium that can be read by the processing unit 110. The storage device 140 includes, for example, non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), or EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory).

[0021] The non-volatile memory of the storage device 140 pre-stores a program PRA that causes the processing device 110 to execute the display method of this disclosure. The non-volatile memory also pre-stores conversion data for mutually converting between the position on the captured image captured by the camera 130 and the position on the display image displayed by the projection device 120. This conversion data is appropriately generated by performing a well-known calibration and stored in the non-volatile memory. Calibration refers to the process of associating the camera coordinate system that defines the position on the captured image captured by the camera 130 with the panel coordinate system that defines the position on the liquid crystal panel in the projection device 120. A specific example of the conversion data is a conversion matrix for mutually converting between the camera coordinate system and the panel coordinate system. The volatile memory of the storage device 140 is used by the processing device 110 as a work area when executing the program PRA. Here, in the case where the projection device 120 is equipped with a DMD instead of a liquid crystal panel, the panel coordinate system is the coordinate system that defines the position on the DMD.

[0022] The processing unit 110 reads the program PRA from non-volatile memory to volatile memory when the projector 10 is powered on, and starts executing the read program PRA. The processing unit 110, operating according to the program PRA, functions as the first acquisition unit 110a, second acquisition unit 110b, first determination unit 110c, second determination unit 110d, and display control unit 110e shown in Figure 7. The first acquisition unit 110a, second acquisition unit 110b, first determination unit 110c, second determination unit 110d, and display control unit 110e shown in Figure 7 are software modules realized by operating the processing unit 110 according to the program PRA. The functions performed by each of the first acquisition unit 110a, second acquisition unit 110b, first determination unit 110c, second determination unit 110d, and display control unit 110e shown in Figure 7 are as follows.

[0023] The first acquisition unit 110a acquires image data output from the camera 130 each time a predetermined range including the display surface SS is imaged by the camera 130. As described above, in this embodiment, the camera 130 images a predetermined range including the display surface SS at a predetermined period. In other words, the first acquisition unit 110a acquires first image data representing a first image of the predetermined range including the display surface SS, and after acquiring the first image data, acquires second image data representing a second image of the predetermined range including the display surface SS.

[0024] The second acquisition unit 110b analyzes the image data acquired by the first acquisition unit 110a to detect bright spots corresponding to the reflected light of the light curtain by the indicator 30 from the captured image data. The second acquisition unit 110b detects in the captured image regions where a predetermined number of pixels, such as 3 pixels or more, belong and the brightness of each pixel is greater than or equal to a predetermined value, as regions corresponding to the bright spots. The bright spots detected by the second acquisition unit 110b from the first captured image are an example of the first bright spots in this disclosure. The bright spots detected by the second acquisition unit 110b from the second captured image are an example of the second bright spots in this disclosure.

[0025] The second acquisition unit 110b acquires a first brightness value representing the brightness of the first bright spot from the first captured image, and a second brightness value representing the brightness of the second bright spot from the second captured image. A specific example of the first brightness value is the average brightness value of pixels belonging to the region corresponding to the first bright spot in the first captured image. A specific example of the second brightness value is the average brightness value of pixels belonging to the region corresponding to the second bright spot in the second captured image. The second acquisition unit 110b also calculates the centroid position of the brightness of the region corresponding to the first bright spot in the first captured image, and acquires this centroid position as the position of the first bright spot in the first captured image. The centroid position of brightness is a value obtained by weighting the X and Y coordinate values ​​of pixels belonging to the region corresponding to the bright spot with the brightness value of the pixel as the weight. The position of the first bright spot in the first captured image is an example of the first position in this disclosure. Furthermore, the second acquisition unit 110b calculates the centroid position of the luminance of the region corresponding to the second bright spot in the second captured image, and acquires this centroid position as the position of the second bright spot in the second captured image. The position of the second bright spot in the second captured image is an example of the second position in this disclosure. In order to improve the stability of bright spot detection, the second acquisition unit 110b may perform bright spot detection after removing the background image obtained by imaging the display surface SS in state ST1 from the captured image.

[0026] The first determination unit 110c determines whether the second luminance value and the second position satisfy the following first condition. The first condition is that the second luminance value is greater than the first luminance value by a first threshold or more, and the second position is closer to the camera 130 than the first position. The second position being closer to the camera 130 than the first position means that the distance between the reference position corresponding to the position of the camera 130 in the panel coordinate system and the third position corresponding to the second position in the panel coordinate system is shorter than the distance between the reference position and the fourth position corresponding to the first position in the panel coordinate system. The reference position can be obtained during the calibration process described above. Specifically, a pattern image for associating the camera coordinate system and the panel coordinate system is projected from the projection device 120 onto the display surface SS, and the above-mentioned conversion data is generated based on the position of each pattern represented in the pattern image included in the third captured image obtained by imaging a predetermined area including this pattern image with the camera 130. The reference position is then obtained by projecting the center of the imaging lens of the camera 130 onto the display surface SS using this conversion data. Furthermore, the third position is obtained by converting the second position to a position in the panel coordinate system using conversion data. The fourth position is obtained by converting the first position to a position in the panel coordinate system using conversion data.

[0027] Figure 8 is a diagram illustrating the positional relationship between the bright spot PA formed by the indicator 30 reflecting IR light LA, the bright spot PB formed by the indicator 30 reflecting IR light LB, and the reference position P0 in the aforementioned state ST3. If the difference between distance HA and distance HB is sufficiently large, bright spots PA and PB will appear as two separate bright spots in the captured image, and the position of bright spot PB will generally be on the straight line connecting the reference position P0 and the position of bright spot PA, as shown in Figure 8. If the difference between distance HA and distance HB is not sufficiently large, bright spots PA and PB will appear as a single bright spot in the captured image, and its brightness will be brighter than the brightness of bright spot PB alone, and its position will be between bright spots PA and PB. In state ST3, the variation in brightness of the bright spots caused by the user moving the indicator 30 along the display surface SS is approximately 20-30%.

[0028] In the image captured in state ST2 described above, only the bright spot PB is visible. The range of change in the brightness of the bright spot when changing from state ST2 to state ST3 is greater than the range of change in the brightness of the bright spot caused by the user moving the indicator 30 along the display surface SS in state ST3. In this embodiment, the first threshold is set to a value greater than the range of change in brightness caused by the user moving the indicator 30 along the display surface SS in state ST3. Therefore, the fact that the second brightness value and the second position satisfy the first condition means that the process of transitioning from state ST2 to state ST3 is in progress. In other words, the first determination unit 110c is provided to determine whether or not the process of transitioning from state ST2 to state ST3 is in progress.

[0029] The second determination unit 110d determines whether the second brightness value and the second position satisfy the following second condition. The second condition is that the first brightness value is brighter than the second brightness value by a second threshold, and the second position is further from the camera 130 than the first position. In this embodiment, the second threshold is the same value as the first threshold. Note that the second threshold may be a different value from the first threshold, as long as it is greater than the brightness fluctuation range caused by the user moving the indicator 30 along the display surface SS in state ST3. A position where the second position is further from the camera 130 than the first position means that the distance between the reference position and the aforementioned third position is longer than the distance between the reference position and the aforementioned fourth position. Satisfying the second condition means that the transition from state ST3 to state ST2 is in progress. In other words, the second determination unit 110d is provided to determine whether the transition from state ST3 to state ST2 is in progress.

[0030] The display control unit 110e controls the display of the drawing image representing the lines drawn by the indicator 30. In this embodiment, if the determination result by the first determination unit 110c is negative and the determination result by the second determination unit 110d is also negative, the display control unit 110e generates image data representing a first display image with the drawing image placed at the aforementioned third position and provides it to the projection device 120. Conversely, if the determination result by the first determination unit 110c is positive, or if the determination result by the second determination unit 110d is positive, the display control unit 110e generates image data representing a second display image without the drawing image placed at the aforementioned third position and provides it to the projection device 120. A positive determination result by the first determination unit 110c means that the transition from state ST2 to state ST3 is in progress, and a positive determination result by the second determination unit 110d means that the transition from state ST3 to state ST2 is in progress. In this embodiment, during the transition from state ST2 to state ST3, and during the transition from state ST3 to state ST2, the drawing image representing the line drawn by the indicator 30 is not displayed. Therefore, no extra lines, namely line segments L2 and L3 in Figure 6, are displayed at the beginning and end of the line drawn using the indicator 30.

[0031] Furthermore, the processing unit 110, which operates according to the program PRA, displays the display method shown in Figure 9 at regular intervals. repetition Execute. As shown in Figure 9, the display method of this embodiment includes a first acquisition process SA110, a second acquisition process SA120, a first determination process SA130, a second determination process SA140, a first display control process SA150, and a second display control process SA160.

[0032] In the first acquisition process SA110, the processing unit 110 functions as a first acquisition unit 110a. In the first acquisition process SA110, the processing unit 110 acquires first image data representing the first captured image, and after acquiring the first image data, acquires second image data representing the second captured image.

[0033] In the second acquisition process SA120, the processing unit 110 functions as the second acquisition unit 110b. In the second acquisition process SA120, the processing unit 110 acquires a first brightness value and a first position by analyzing the first image data. In addition, in the second acquisition process SA120, the processing unit 110 acquires a second brightness value and a second position by analyzing the second image data.

[0034] In the first determination process SA130, the processing unit 110 functions as the first determination unit 110c. In the first determination process SA130, the processing unit 110 determines whether the second brightness value and the second position satisfy the first condition. If the second brightness value and the second position satisfy the first condition, the determination result of the first determination process SA130 is "Yes", i.e., affirmative. If the determination result of the first determination process SA130 is affirmative, the processing unit 110 executes the first display control process SA150 and terminates the display method. If the second brightness value and the second position do not satisfy the first condition, the determination result of the first determination process SA130 is "No", i.e., negative. If the determination result of the first determination process SA130 is negative, the processing unit 110 executes the second determination process SA140.

[0035] In the second determination process SA140, the processing unit 110 functions as a second determination unit 110d. In the second determination process SA140, the processing unit 110 determines whether the second brightness value and the second position satisfy the second condition. If the second brightness value and the second position satisfy the second condition, the determination result of the second determination process SA140 is "Yes", i.e., affirmative. If the determination result of the second determination process SA140 is affirmative, the processing unit 110 executes the first display control process SA150 and terminates this display method. If the second brightness value and the second position do not satisfy the second condition, the determination result of the second determination process SA140 is "No", i.e., negative. If the determination result of the second determination process SA140 is negative, the processing unit 110 executes the second display control process SA160 and terminates this display method. end do.

[0036] In the first display control process SA150 and the second display control process SA160, the processing unit 110 functions as a display control unit 110e. In the first display control process SA150, the processing unit 110 generates image data representing a first display image that does not include a drawing image at a third position and provides it to the projection device 120. In the second display control process SA160, the processing unit 110 generates image data representing a second display image that includes a drawing image at a third position and provides it to the projection device 120.

[0037] In this embodiment, whether or not a line has started to be drawn and whether or not a line has ended is determined based on the magnitude of the change in the brightness of the bright spot as well as the magnitude of the change in the position of the bright spot. Therefore, the projector 10 of this embodiment can more accurately detect the touch of the indicator 30 on the display surface SS compared to the technology disclosed in Patent Document 1, and the display of extra lines at the start and end of lines drawn using the indicator 30 is avoided.

[0038] 2. Transformation The above embodiment can be modified as follows. (1) If it is acceptable for an extra line to be displayed at the endpoint of the drawn line, the second determination process SA140 may be omitted. In the display method in which the second determination process SA140 is omitted, if the determination result of the first determination process SA130 is negative, the second display control process SA160 is executed. If it is acceptable for an extra line to be displayed at the starting point of the drawn line, the first determination process SA130 may be omitted, in which case the second determination process SA140 may be executed following the second acquisition process SA120. The display method of this disclosure may also include a process for acquiring a reference position, more specifically calibration. In the above embodiment, it was determined whether the first condition and the second condition were met based on the position of the first bright point, the position of the second bright point, and the position corresponding to the camera 130 in the panel coordinate system. However, it is also possible to determine whether the first condition and the second condition were met based on the position of the first bright point, the position of the second bright point, and the position corresponding to the camera 130 in the camera coordinate system. This is because the positional relationship between the position of the first bright spot, the position of the second bright spot, and the position corresponding to camera 130 is invariant regardless of whether it is in the panel coordinate system or the camera coordinate system.

[0039] (2) In the above embodiment, the camera 130 was included in the projector 10, but the camera 130 may not be included in the projector 10 and may be a separate component. Although the above embodiment was an example of applying the present disclosure to a projector, the scope of application of the present disclosure is not limited to projectors, and may also be a flat panel display. In short, the present disclosure can be applied to any display device that displays an image corresponding to the user's writing action on the display surface. Here, in the case of a configuration in which a flat panel display is used instead of the projector 10, the panel coordinate system is a coordinate system that defines the position on the display panel in which the flat panel display displays an image.

[0040] (3) In the above embodiment, the first acquisition unit 110a, the second acquisition unit 110b, the first determination unit 110c, the second determination unit 110d, and the display control unit 110e were software modules. However, one or more of the first acquisition unit 110a, the second acquisition unit 110b, the first determination unit 110c, the second determination unit 110d, and the display control unit 110e, or all of them, may be hardware modules such as ASICs (Application Specific Integrated Circuits). Even if one or more of the first acquisition unit 110a, the second acquisition unit 110b, the first determination unit 110c, the second determination unit 110d, and the display control unit 110e, or all of them, are hardware modules, the same effects as in the above embodiment will be achieved.

[0041] (4) Program PRA may be manufactured as a standalone product and may be provided for a fee or free of charge. Specific ways of providing Program PRA include providing it by writing it to a computer-readable recording medium such as flash ROM, or by providing it by downloading it via a telecommunications line such as the Internet. By operating a general computer in accordance with Program PRA provided in these ways, it becomes possible to have the computer execute the display method of the Disclosure.

[0042] 3. Summary of this disclosure This disclosure is not limited to the embodiments and modifications described above, and can be implemented in various forms without departing from its spirit. For example, this disclosure can also be implemented in the following forms. The technical features in the embodiments described above that correspond to the technical features in each of the forms described below can be replaced or combined as appropriate in order to solve some or all of the problems of this disclosure or to achieve some or all of the effects of this disclosure. Furthermore, if such technical features are not described as essential in this specification, they can be deleted as appropriate. A summary of this disclosure is provided below.

[0043] (Note 1) A display method according to one aspect of the present disclosure includes: acquiring a first image captured by a camera of an area including a display surface; acquiring a second image captured by the camera of an area including the display surface after acquiring the first image; acquiring a first luminance value indicating the brightness of a first bright spot included in the first image captured by reflected light from an indicator for writing on the display surface, and a first position indicating the coordinates of the first bright spot in the first image captured; and acquiring a second luminance value indicating the brightness of a second bright spot included in the second image captured by reflected light from the indicator, and The method includes obtaining a second position indicating the coordinates of the second bright spot in the second captured image, and if the first condition is met, such that the second brightness value is greater than or equal to a first threshold and the second position is closer to the camera than the first position, displaying a first display image on the display surface that does not include a drawing image representing the line by the indicator at a third position corresponding to the second position in the image to be displayed on the display surface, and if the first condition is not met, displaying a second display image including the drawing image at the third position on the display surface.

[0044] The fulfillment of the first condition means that the user is bringing the indicator closer to the display surface in order to draw letters or figures. In the display method described in (Note 1), when the first condition is fulfilled, a first display image that does not include a drawing image representing the line drawn by the indicator is displayed on the display surface, so that the touch of the indicator on the display surface can be correctly detected and the display of extra lines connected to the starting point of the line drawn by the touch of the indicator on the display surface is avoided.

[0045] (Note 2) A more preferred embodiment of the display method is the display method described in Appendix 1, which further includes displaying the first display image on the display surface when the first brightness value is brighter than the second brightness value by a second threshold and the second position is further from the camera than the first position, thus satisfying the second condition.

[0046] The second condition being met means that the user drawing letters or figures using the indicator is moving the indicator away from the display surface to finish drawing. In the display method described in (Note 2), when the second condition is met, the first display image, which does not include the drawing image representing the lines drawn by the indicator, is displayed on the display surface, thus avoiding the display of extra lines connected to the endpoint of the lines drawn by touching the indicator to the display surface.

[0047] (Note 3) A more preferred embodiment of the display method is the display method described in (Note 1) or (Note 2), wherein displaying the first display image on the display surface includes the display device displaying the first display image on the display surface using a display panel provided by the display device, and further includes obtaining a reference position corresponding to the position of the camera in a panel coordinate system that defines the position on the display panel, wherein the second position being closer to the camera than the first position is defined as the distance from the reference position to the third position being shorter than the distance from the reference position to the fourth position corresponding to the first position in the panel coordinate system.

[0048] According to the display method described in (Note 3), the reference position corresponding to the camera position in the panel coordinate system , Pa Based on the fourth position corresponding to the first position in the Nell coordinate system, and the third position corresponding to the second position in the panel coordinate system, it becomes possible to determine whether the first condition is met.

[0049] (Note 4) A more preferred embodiment of the display method is the display method described in (Note 1) or (Note 2), wherein displaying the first display image on the display surface includes the display device displaying the first display image on the display surface using a display panel provided by the display device, and further includes obtaining a reference position corresponding to the position of the camera in a panel coordinate system that defines the position on the display panel, wherein the second position being further from the camera than the first position is defined as the distance from the reference position to the third position being longer than the distance from the reference position to the fourth position corresponding to the first position in the panel coordinate system.

[0050] According to the display method described in (Note 4), the reference position corresponding to the camera position in the panel coordinate system , Pa Based on the fourth position corresponding to the first position in the Nell coordinate system, and the third position corresponding to the second position in the panel coordinate system, it becomes possible to determine whether the second condition is met.

[0051] (Note 5) A more preferred embodiment of the display method is the display method described in (Appendix 3) or (Appendix 4), which includes obtaining the reference position by displaying a pattern image on the display surface, obtaining a third captured image of a region including the pattern image captured by the camera, and obtaining the reference position by associating a camera coordinate system that defines a position on the captured image by the camera based on the pattern image included in the third captured image with the panel coordinate system.

[0052] According to the display method described in (Appendix 5), it becomes possible to obtain a reference position based on a third captured image obtained by capturing an image of the display surface on which the pattern image is projected.

[0053] (Note 6) A display system according to one aspect of the present disclosure includes: a camera that images an area including a display surface; acquiring a first image captured by the camera on the area; acquiring a second image captured by the camera on the area after acquiring the first image; acquiring a first luminance value indicating the brightness of a first bright spot included in the first image based on reflected light from an indicator for drawing lines on the display surface; a first position indicating the coordinates of the first bright spot in the first image; and a second luminance value indicating the brightness of a second bright spot included in the second image based on reflected light from the indicator. A display device that performs the following: obtaining a second position indicating the coordinates of the second bright spot in a second captured image; if the second brightness value is greater than or equal to a first threshold and the second position is closer to the camera than the first position, displaying a first display image on the display surface that does not include a drawing image representing the line by the indicator at a third position corresponding to the second position in the image to be displayed on the display surface; and if the first condition is not met, displaying a second display image including the drawing image at the third position on the display surface.

[0054] According to the display system described in (Appendix 6), similar to the display method described in (Appendix 1), when the first condition is met, a first display image is displayed that does not include a drawing image representing a line drawn by the indicator. This allows for accurate detection of the indicator's touch on the display surface and avoids the display of extra lines connected to the starting point of the line drawn by the indicator's touch on the display surface. [Explanation of symbols]

[0055] 1...Display system, 10, 10A, 10B...Projector, 20A, 20B...Irradiation device, 30...Indicator, 110...Processing device, 110a...First acquisition unit, 110b...Second acquisition unit, 110c...First determination unit, 110d...Second determination unit, 110e...Display control unit, 120...Projection device, 130...Camera, 140...Storage device, PRA...Program, SS...Display surface.

Claims

1. The process involves acquiring a first image captured by a camera, which includes the display surface, After acquiring the first image, a second image is acquired by capturing the area including the display surface with the camera. The acquisition of a first brightness value indicating the brightness of a first bright spot included in the first captured image, and a first position indicating the coordinates of the first bright spot in the first captured image, based on reflected light from an indicator for drawing lines on the display surface, The method involves obtaining a second luminance value indicating the brightness of a second bright spot included in the second captured image based on the reflected light from the indicator, and a second position indicating the coordinates of the second bright spot in the second captured image. If the second brightness value is greater than or equal to a first threshold than the first brightness value, and the second position is closer to the camera than the first position, then the first display image, which does not include a drawing image representing the line by the indicator, is displayed on the display surface at a third position corresponding to the second position in the image displayed on the display surface. If the first condition is not met, a second display image including the drawing image is displayed on the display surface at the third position, Display methods, including those mentioned above.

2. The first display image is displayed on the display surface when the second condition is met, namely, the first brightness value is greater than or equal to a second threshold than the second brightness value, and the second position is further from the camera than the first position. The display method according to claim 1, further comprising:

3. Displaying the first display image on the display surface is The display device displays the first display image on the display surface using the display panel provided by the display device. Includes, The method further includes obtaining a reference position corresponding to the position of the camera in a panel coordinate system that defines the position of the display panel, The second position being closer to the camera than the first position is defined as the distance from the reference position to the third position being shorter than the distance from the reference position to the fourth position corresponding to the first position in the panel coordinate system. The display method according to claim 1.

4. Displaying the first display image on the display surface is The display device displays the first display image on the display surface using the display panel provided by the display device. Includes, The method further includes obtaining a reference position corresponding to the position of the camera in a panel coordinate system that defines the position of the display panel, The second position being further from the camera than the first position is defined as the distance from the reference position to the third position being longer than the distance from the reference position to the fourth position corresponding to the first position in the panel coordinate system. The display method according to claim 2.

5. Obtaining the aforementioned reference position means Displaying a pattern image on the aforementioned display surface, A third image is obtained by capturing the region including the aforementioned pattern image with the camera, The reference position is obtained by associating the camera coordinate system, which defines the position on the image captured by the camera based on the pattern image included in the third captured image, with the panel coordinate system, A display method according to claim 3 or claim 4, including the method described in claim 4.

6. A camera that captures an image of the area including the display surface, To acquire a first image captured by the camera within the aforementioned range, After acquiring the first image, a second image is acquired by capturing the range with the camera. The acquisition of a first brightness value indicating the brightness of a first bright spot included in the first captured image, and a first position indicating the coordinates of the first bright spot in the first captured image, based on reflected light from an indicator for drawing lines on the display surface, The method involves obtaining a second luminance value indicating the brightness of a second bright spot included in the second captured image based on the reflected light from the indicator, and a second position indicating the coordinates of the second bright spot in the second captured image. If the second brightness value is greater than or equal to a first threshold than the first brightness value, and the second position is closer to the camera than the first position, then the first display image, which does not include a drawing image representing the line by the indicator, is displayed on the display surface at a third position corresponding to the second position in the image displayed on the display surface. If the first condition is not met, a display device that performs the following: display a second display image including the drawing image on the display surface at the third position; Display system.

Images