Display system, display device, and recording medium containing the program.
By using a learned model generated through machine learning in the display system, the image content and display method are determined based on the relationship between behavioral features and risk level, thus solving the problem of ineffective reminders in existing technologies and achieving appropriate and effective attention reminders.
Patent Information
- Application Number
- CN202310736322.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-06-22
- Filing Date
- 2023-06-20
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-06-20
AI Technical Summary
Existing display systems are unable to effectively provide appropriate attention reminders and have not fully considered the effect of warning displays.
Using a display device, a shooting device, and a processing device, a learned model is generated through machine learning. Based on the relationship between the captured behavioral features and the degree of danger, the image content and display method are determined to minimize the danger to people passing through the display location.
It enables appropriate attention reminders for people passing through the displayed location, reduces the decrease in warning or attention effectiveness due to habits and other reasons, and improves the effectiveness of warning or attention.
Smart Images

Figure CN117278725B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to display systems, display devices, and recording media containing programs. Background Technology
[0002] Conventional display systems that display images corresponding to human behavior are known. For example, the system described in Patent Document 1 includes a display device, a camera that captures images of a person, and a server that predicts the person's future behavior based on the images captured by the camera and displays a warning image generated based on the prediction results on the display device.
[0003] Patent Document 1: Japanese Patent Application Publication No. 2014-123277
[0004] In the system described in Patent Document 1, the effect of warning display is not fully considered, and therefore, it is sometimes impossible to provide appropriate attention reminders to the target person. Summary of the Invention
[0005] One aspect of the display system disclosed herein includes a display device that displays an image based on content information at a display location; a first capturing device; a storage device that stores a learned model obtained by machine learning of a correspondence between pre-passage behavior data representing feature quantities of a person's behavior before passing the display location, captured by the first capturing device, content information used in the image, and the danger level of the person who passed the display location; and a processing device that performs the following processing: obtaining first pre-passage behavior data representing feature quantities of the behavior of the first person before passing the display location, based on a capture result obtained by capturing a first person before passing the display location using the first capturing device; and determining, using the first pre-passage behavior data and the learned model, first content information to be used in a first image corresponding to the first pre-passage behavior data.
[0006] One aspect of this disclosure is a display device that displays an image based on content information at a display location. The display device includes: a storage device that stores a learned model obtained by machine learning of pre-passage behavior data representing feature quantities of a person's behavior before passing the display location, content information used in the image, and a correspondence between the risk level of the person who passed the display location; and a processing device that performs the following processing: obtaining first pre-passage behavior data representing feature quantities of the behavior of the first person before passing the display location based on an image taken of a first person before passing the display location; and using the first pre-passage behavior data and the learned model, determining first content information to be used in a first image corresponding to the first pre-passage behavior data.
[0007] One aspect of this disclosure includes a recording medium containing a program that causes a computer to perform the following processing: obtaining first pre-passage behavior data based on a photograph of a first person before the display position of a first image based on first content information, representing a feature quantity of the behavior of the first person before the display position; and determining the first content information used in the first image using a learned model obtained by machine learning of the correspondence between the feature quantity of the behavior of the person before the display position, the content information used in the image, and the danger level of the person after the display position, and the first pre-passage behavior data. Attached Figure Description
[0008] Figure 1 This is a schematic diagram of the display system according to the first embodiment.
[0009] Figure 2 This is a block diagram of the display system according to the first embodiment.
[0010] Figure 3 This is a flowchart illustrating the operation of the display system according to the first embodiment.
[0011] Figure 4 This is a diagram used to illustrate the learned model in the first embodiment.
[0012] Figure 5 This is a diagram used to illustrate the machine learning process of generating a learned model in the first embodiment.
[0013] Figure 6 This is a schematic diagram of the display system according to the second embodiment.
[0014] Figure 7 This is a schematic diagram of the display system according to the third embodiment.
[0015] Figure 8 This is a block diagram of the display system according to the fourth embodiment.
[0016] Figure 9 This is a flowchart illustrating the operation of the display system according to the fourth embodiment.
[0017] Figure 10 This is a diagram used to illustrate the learned model in the fourth embodiment.
[0018] Figure 11 This is a diagram used to illustrate the machine learning process of generating a learned model in the fourth embodiment.
[0019] Figure 12 This is a graph illustrating an example of a score based on the preceding data for each action.
[0020] Label Explanation
[0021] 1 Display system; 1A Display system; 1B Display system; 1C Display system; 10 Display device; 11 Projection device; 12 Playback device; 20 Shooting device; 21 First shooting device; 22 Second shooting device; 30 Control device (computer); 30A Control device (computer); 30B Control device (computer); 30C Control device (computer); 31 Communication device; 32 Storage device; 33 Processing device; 33a Acquisition unit; 33b Decision unit; 33c Display control unit; 33d Relearning unit; 33e Acquisition unit; 33f Decision unit; 33g Relearning unit; 100 Learning processing unit; 100C Learning processing unit; AL path; AL1 path; AL2 path; AL3 path; DA hazard level; DA1 forward pass Data; DA2 post-passage behavior data; DC content information; DC1 content information; DE environmental data; DR travel direction; DS1 first shot data; DS2 second shot data; DST dataset; DST1 dataset; FF ground; G image; H person; H_a person; H_b person; PD display position; PG program; PG1 program; PG2 program; PJ learned model; PJC learned model; RE1 first region; RE2 second region; S1 step; S10 step; S11 step; S11A step; S12 step; S13 step; S2 step; S3 step; S4 step; S4A step; S5 step; S6 step; S7 step; S8 step; S9 step; S9A step; ST steps. Detailed Implementation
[0022] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the dimensions and scales of the parts in the drawings differ appropriately from actual dimensions, and some parts are shown schematically for ease of understanding. Furthermore, the scope of the present disclosure is not limited to these embodiments unless otherwise specified in the following description.
[0023] 1. First Implementation Method
[0024] 1-1. Overview of the Display System
[0025] Figure 1 This is a schematic diagram of the display system 1 according to the first embodiment. The display system 1 is a system that alerts the person H by displaying an image G corresponding to the behavior of the person H at the display position PD.
[0026] exist Figure 1 In the example shown, display system 1 uses image G to draw attention to person H traveling along path AL. Here, a step ST is located midway along the length of path AL on the ground FF. The display position PD is positioned on the ground FF adjacent to step ST in the direction of travel DR of person H. Figure 1 The image G illustrates a reminder to person H to pay attention to step ST. The direction of travel DR is a direction along the length of the path AL.
[0027] Furthermore, the width or shape of the pathway AL is not limited to Figure 1 The example shown can be arbitrary. Furthermore, the display position PD is not limited to... Figure 1 The example shown could be a location away from the step ST, or a location set on the screen of the passage AL, on the left or right walls of the passage AL, or on the ceiling. The display position PD is, for example, the center of the image G.
[0028] The display system 1 includes a display device 10, a first imaging device 21, a second imaging device 22, and a control device 30. Hereinafter, according to... Figure 1 Let me give you a brief explanation of them.
[0029] Display device 10 is a projector that displays image G at display position PD under the control of control device 30. (This will be discussed later based on...) Figure 2 The detailed structure of the display device 10 is described.
[0030] The first shooting device 21 is a camera with a camera element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor that captures images of a person H passing in front of the display position PD.
[0031] The second shooting device 22 is a camera equipped with an imaging element such as a CCD image sensor or a CMOS image sensor to capture images of a person H passing behind the display position PD. Alternatively, the first shooting device 21 can also function as the second shooting device 22.
[0032] Based on the imaging results of the first imaging device 21, the control device 30 determines the display content and display method of the image G corresponding to the behavior of the person H before passing the display position PD, and uses the display device 10 to display the image G with the determined display content at the display position PD using the determined display method. Here, in determining the display content and display method of the image G, a learned model PJ (described later) obtained by machine learning the prescribed correspondence in a manner that minimizes the risk to the person H who passed the display position PD is used. Furthermore, the control device 30 uses the imaging results of the second imaging device 22 to relearn the learned model PJ (described later). The detailed structure of the control device 30 will be discussed later based on… Figure 2 Please provide an explanation.
[0033] In the above-described display system 1, the display content and display method of image G are determined by using a fully learned model PJ obtained by machine learning on the prescribed correspondence in a manner that minimizes the risk to the person H who has passed the display position PD. Therefore, effective attention reminders based on image G can be provided.
[0034] 1-2. Structure of the display system
[0035] Figure 2 This is a block diagram of the display system 1 according to the first embodiment. Figure 2 In addition to the electrical structure of the display system 1, a diagram showing the path AL viewed from a vertically upward position is also shown. As described above, the display system 1 includes a display device 10, a first imaging device 21, a second imaging device 22, and a control device 30. Hereinafter, according to... Figure 2 The various parts of display system 1 will be described in detail in turn.
[0036] like Figure 2 As shown, the display device 10 includes a projection device 11, which serves as an optical device, and a playback device 12.
[0037] The projection device 11 is a mechanism that projects an image G onto a display position PD under the control of the control device 30. Although not shown, the projection device 11 includes, for example, an image processing circuit, a light source, a light modulation device, and a projection optical system.
[0038] The image processing circuit of the projection device 11 is a circuit that uses image data from the content information DC of the control device 30 to generate an image signal for driving the optical modulation device. Specifically, the image processing circuit has a frame memory, in which the image data from the content information DC is expanded and various processes such as resolution transformation processing, size adjustment processing, and distortion correction processing are performed appropriately to generate an image signal.
[0039] The light source of the projection device 11 includes, for example, a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source. This light source emits, for example, white light, or red, green, and blue light respectively. In the case where the light source emits white light, the light emitted from the light source passes through an integrator optical system (not shown) to reduce the brightness distribution deviation, and then is separated into red, green, and blue light by a color separation optical system (not shown) before being incident on the light modulation device of the projection device 11.
[0040] The light modulation device of the projection device 11 includes three light modulation elements corresponding to the red, green, and blue light mentioned above. These three light modulation elements include, for example, a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (digital mirror device). Based on the image signal from the image processing circuit of the projection device 11, the three light modulation elements modulate the red, green, and blue light respectively to generate image light of each color. This image light of each color is then synthesized by a color-combining optical system (not shown) to form full-color image light.
[0041] The projection optical system of the projection device 11 images and projects the full-color image light onto the projection surface. This projection optical system is, for example, an optical system including a projection lens. In addition to the projection lens, the projection optical system may also include, for example, a zoom lens or a focusing lens.
[0042] The playback device 12 is a speaker that plays sound to the vicinity of the display position PD using sound data from the content information DC of the control device 30. The sound-emitting device 12 may also include circuitry such as an audio amplifier for reproducing the sound data. Alternatively, the playback device 12 may be separately mounted outside the display device 10.
[0043] The first imaging device 21 generates first imaging data DS1 by photographing a person H in the first region RE1. The first imaging data DS1 can be static image data or dynamic image data, but dynamic image data is preferred because it can appropriately extract the characteristic quantities of the person H's behavior. The form of the first imaging data DS1 is not particularly limited and is arbitrary, for example, it can be a known static image or dynamic image data form.
[0044] exist Figure 2 In the example shown, the first region RE1 is a region extending along the length of the path AL, behind the display position PD in the direction of human H's movement. The length of the first region RE1 along the length of the path AL is sufficient to extract the characteristic quantity of human H's behavior from the imaging results of the first imaging device 21, for example, more than 1m and less than 2m.
[0045] The second imaging device 22 generates second imaging data DS2 by photographing a person H in a second region RE2, which is different from the first region RE1. The second imaging data DS2 can be static image data or dynamic image data, but dynamic image data is preferred because it can appropriately extract the behavioral features of person H. The form of the second imaging data DS2 is not particularly limited and can be arbitrary, for example, it can be a known static image or dynamic image data form.
[0046] exist Figure 2 In the example shown, the second region RE2 is a region extending along the length of the path AL, in front of the walking direction DR of the person H relative to the display position PD. The length of the second region RE2 along the length of the path AL is sufficient to extract the characteristic quantity of the person H's behavior from the imaging results of the second imaging device 22, for example, more than 1m and less than 2m. In addition, when the imaging range of the first imaging device 21 includes both the first region RE1 and the second region RE2, the first imaging device 21 can also serve as the second imaging device 22.
[0047] Control device 30 is a computer that executes program PG1. For example... Figure 2 As shown, the control device 30 includes a communication device 31, a storage device 32, and a processing device 33. They are communicatively connected to each other via a common bus.
[0048] The communication device 31 is a device capable of communicating with the display device 10, the first shooting device 21, and the second shooting device 22 via wired or wireless means, respectively. Specifically, the communication device 31 has interface circuits for communicating with the display device 10, interface circuits for communicating with the first shooting device 21, and interface circuits for communicating with the second shooting device 22. For example, the communication device 31 includes wired communication devices such as wired LAN (Local Area Network), USB (Universal Serial Bus), and HDMI (High Definition Multimedia Interface), as well as wireless communication devices such as LPWA (Low Power Wide Area), wireless LAN including Wi-Fi, and Bluetooth. "Wi-Fi," "HDMI," and "Bluetooth" are registered trademarks.
[0049] Storage device 32 is a device for storing various programs executed by processing device 33 and various information processed by processing device 33. Storage device 32 is, for example, a hard disk drive or a semiconductor memory. In addition, some or all of the information stored in storage device 32 may be pre-stored or may be obtained from an external device via the communication device 31.
[0050] The storage device 32 stores program PG1, learned model PJ, preceding action data DA1, following action data DA2, first shot data DS1, second shot data DS2, and content information DC.
[0051] Program PG1 is a program used to enable the processing device 33 to function as the various functional units described later.
[0052] The preceding behavior data DA1 represents the characteristic data of the person H's behavior before passing the display position PD. The following behavior data DA2 represents the characteristic data of the person H's behavior after passing the display position PD.
[0053] The content information DC contains image data representing the content displayed as an image G on the display device 10. This image data represents an image that draws the attention of the person H. The form of this image data is not particularly limited; for example, it can be in the form of known dynamic data or static image data. Here, in addition to the image data, the content information DC also contains information related to the display method, such as the position, time, hue, and brightness of the displayed image data.
[0054] In this embodiment, the content information DC includes, in addition to image data, sound data reproduced by the playback device 12. This sound data represents a sound that draws attention to a person (H). The form of this sound data is not particularly limited; for example, it can be a known sound data format.
[0055] The learned model PJ is an estimated model obtained by machine learning the correspondence between the features of the behavior of person H before passing through the display location PD, the content information DC used in the image G, and the risk of person H who passed through the display location PD in a way that minimizes the risk of person H who passed through the display location PD.
[0056] The processing device 33 is a processing device that has the function of controlling the operation of various parts of the control device 30 and processing various types of data. The processing device 33 includes, for example, one or more processors such as a CPU (Central Processing Unit). In addition, part or all of the processing device 33 can be implemented by hardware such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array).
[0057] The processing device 33 functions as an acquisition unit 33a, a decision unit 33b, a display control unit 33c, and a relearning unit 33d by executing program PG1.
[0058] The acquisition unit 33a acquires pre-passage behavior data DA1 based on the first shooting data DS1 or acquires post-passage behavior data DA2 based on the second shooting data DS2.
[0059] The decision unit 33b uses the prior behavior data DA1 and the learned model PJ to determine the content information DC, so as to minimize the risk of the person H who has passed the display position PD.
[0060] The display control unit 33c causes the display device 10 to display an image G based on the content information DC determined by the decision unit 33b.
[0061] The relearning unit 33d uses the post-behavior data DA2 to relearn the learned model PJ. Here, as described later, the risk level of person H after passing the display position PD can be calculated based on the post-behavior data DA2. Therefore, the relearning unit 33d uses a combination of the post-behavior data DA2 or the risk level calculated based on the post-behavior data DA2, the pre-behavior data DA1, and the content information DC for relearning.
[0062] 1-3. Display System Operations
[0063] Figure 3 This is a flowchart illustrating the operation of the display system 1 according to the first embodiment. In the display system 1, firstly, as... Figure 3 As shown, in step S1, the acquisition unit 33a acquires the first image data DS1. The acquired first image data DS1 is stored in the storage device 32.
[0064] Next, in step S2, the acquisition unit 33a determines whether a person H exists in the first region RE1. This determination is performed, for example, by image processing that identifies a person based on the first captured data DS1. If no person H exists in the first region RE1 (step S2: No), the acquisition unit 33a returns to step S1. The person H present in the first region RE1 is equivalent to the first person.
[0065] If person H is present in region RE1 (step S2: Yes), in step S3, the acquisition unit 33a acquires the preceding behavior data DA1. This acquisition can be achieved, for example, by directly acquiring the first captured data DS1 during the period when person H is present in region RE1 as the preceding behavior data DA1, or by performing appropriate processing on the first captured data DS1 during the period when person H is present in region RE1 as required for applying to the learned model PJ. The preceding behavior data DA1 acquired in step S3 is equivalent to the first preceding behavior data.
[0066] Next, in step S4, the decision unit 33b determines the content information DC using the preceding data DA1 and the learned model PJ. This decision is made by using the preceding data DA1 as input data and obtaining the content information DC as the output data of the learned model PJ. Alternatively, the content information DC can be determined by performing appropriate post-processing on the output data of the learned model PJ. The content information DC determined in step S4 corresponds to the first content information.
[0067] Next, in step S5, the display control unit 33c causes the display device 10 to display an image G based on the content information DC determined by the decision unit 33b.
[0068] Next, in step S6, the acquisition unit 33a acquires the second image data DS2. The acquired second image data DS2 is stored in the storage device 32.
[0069] Next, in step S7, the acquisition unit 33a determines whether a person H exists in the second region RE2. This determination is made, for example, by image processing that identifies a person based on the second captured data DS2. If no person H exists in the second region RE2 (step S7: No), the acquisition unit 33a returns to step S6.
[0070] If person H is present in region RE2 (step S7: Yes), in step S8, the acquisition unit 33a acquires the behavior data DA2 after passage. This acquisition can be achieved, for example, by directly acquiring the second captured data DS2 during the period when person H is present in region RE2 as the behavior data DA2 after passage, or by performing appropriate processing on the second captured data DS2 during the period when person H is present in region RE2 as required for applying to the learned model PJ.
[0071] Next, in step S9, the relearning unit 33d associates the passed post-beginning data DA2 with its corresponding passed pre-beginning data DA1 and content information DC, and stores it as history data in the storage device 32.
[0072] Next, in step S10, the relearning unit 33d determines whether relearning is needed. This determination is made, for example, by checking whether the history data has reached a predetermined amount or whether the history data storage period has reached a predetermined period. Alternatively, relearning can be performed at regular intervals using data from the previous predetermined period. This prevents the reduction in effectiveness caused by human habits. For example, a learned model PJ can be used during the history data accumulation period, i.e., one week. After one week, the relearning unit 33d associates the post-passage behavior data DA2 accumulated in that week with its corresponding pre-passage behavior data DA1 and content information DC, and stores this as history data in the storage device 32.
[0073] If it is determined that relearning is needed (step S10: Yes), in step S11, the relearning unit 33d performs relearning of the learned model PJ and then moves to step S12. On the other hand, if it is determined that relearning is not needed (step S10: No), the relearning unit 33d does not perform relearning of the learned model PJ and moves to step S12.
[0074] In step S12, the processing device 33 determines whether there is an end instruction. If there is no end instruction (step S12: No), it returns to step S1. On the other hand, if there is an end instruction (step S12: Yes), the processing ends.
[0075] 1-4. Model completed
[0076] Figure 4 This diagram illustrates the learned model PJ in the first embodiment. The learned model PJ is an estimated model that outputs content information DC based on input from preceding data DA1. Specifically, the learned model PJ is implemented by having the processing device 33 execute a program that generates content information DC based on the preceding data DA1 and a combination of multiple coefficients applied to that operation. This program is, for example, a program module constituting artificial intelligence software. These multiple coefficients are, for example, set using deep learning of the dataset DST, described later. Figure 4 In the example shown, the learned model PJ is a mathematical model such as a deep neural network with input layers, output layers, and intermediate layers. Furthermore, the number of intermediate layers is not limited to... Figure 4 The examples shown are arbitrary.
[0077] Figure 5 This is a diagram illustrating the machine learning process used to generate the learned model PJ in the first embodiment. The generation of the learned model PJ is as follows: Figure 5As shown, machine learning is performed using a learning device with a learning processing unit 100. Multiple datasets DST are used for this machine learning. Each dataset DST includes pre-passage behavior data DA1, its corresponding content information DC, and a risk level based on post-passage behavior data DA2. The pre-passage behavior data DA1 and content information DC in the dataset DST can be generated using a device different from the control device 30 described above.
[0078] The dataset DST uses the risk level DA of a person H after passing through the display location PD, the displayed content information DC, and their corresponding pre-passage behavior data DA1. The risk level DA of a person H after passing through the display location PD can be calculated based on the post-passage behavior data DA2. For example, the risk level DA is represented by the difference between the feature quantity of the normal behavior of the person H after passing through the display location PD and the feature quantity shown in the post-passage behavior data DA2. The larger this difference, the higher the risk level DA of the person H after passing through the display location PD. Normal behavior is pre-defined statistically considered safe behavior data. The number of times normal behavior is defined can be one or more, or the differences between the feature quantity of multiple normal behaviors and the feature quantity represented by the post-passage behavior data DA2 can be calculated separately, and the maximum value of the difference is taken as the risk level DA of the person H after passing through the display location PD.
[0079] The learning processing unit 100 sets multiple coefficients of the learned model PJ using taught machine learning on multiple datasets DST. Specifically, the learning processing unit 100 updates the multiple coefficients of the learned model PJ to reduce the difference between the content information DC1 output by the temporarily learned model PJ based on the input of the preceding action data DA1 in the dataset DST and the content information DC with the lowest risk contained in the dataset DST. For example, the learning processing unit 100 repeatedly updates the multiple coefficients of the learned model PJ using the error backpropagation method to minimize the evaluation function representing the difference between the content information DC and the content information DC1. Based on the potential trend between the preceding action data DA1, the content information DC, and the risk DA in the multiple datasets DST, the learned model PJ outputs statistically appropriate content information DC for the unknown preceding action data DA1. As a result, the image data with the content information DC changed according to the preceding action data DA1, or the display method such as the position, time, brightness, and hue of the image G displayed based on the image data, can be changed, resulting in a lower risk compared to not displaying the image G based on the image data. Furthermore, when deciding on the content information to display, random content can also be displayed at a certain proportion. This allows the relationship between preceding behavior data DA1, content information DC, and hazard level DA to be added to the dataset DST, enabling the collection of necessary data based on the location or period of the setup and the selection of optimal content.
[0080] As described above, the display system 1 includes a display device 10, a first imaging device 21, a storage device 32, and a processing device 33. The display device 10 displays an image G based on content information DC at a display position PD. The storage device 32 stores a learned model PJ, which is obtained by machine learning of the correspondence between pre-passage behavior data DA1, representing the characteristic quantity of the behavior of a person H before passing the display position PD captured by the first imaging device 21, the content information DC used in the image G, and the danger level of the person H who passed the display position PD. The processing device 33, based on the imaging result obtained by capturing the first person H before passing the display position PD using the first imaging device 21, obtains the first pre-passage behavior data DA1, representing the characteristic quantity of the behavior of the first person H before passing the display position PD, and uses the first pre-passage behavior data DA1 and the learned model PJ to determine the first content information DC to be used in the first image G corresponding to the first pre-passage behavior data DA1.
[0081] In the above-described display system 1, the content information DC used in the image G displayed by the display device 10 is determined using the preceding behavior data DA1 and the learned model PJ in a manner that minimizes the risk to the person H who has passed the display position PD. Therefore, effective attentional reminders can be provided through the image G. Here, in the learned model PJ, this correspondence is machine-learned in a manner that minimizes the risk to the person H who has passed the display position PD. Therefore, by using such a learned model PJ, the effective content information DC that minimizes the risk to the person H who has passed the display position PD can be determined. As a result, appropriate attentional reminders can be provided to the person H who has passed the display position PD.
[0082] As described above, the processing device 33 uses the first preceding data DA1 and the learned model PJ to determine at least one of the following: display position PD, display time of the first image G, and display color or brightness of the first image G. For example, the content information DC includes not only image data but also setting information related to at least one of the display position PD, display time of image G, and display color or brightness of image G. The processing device 33 uses the preceding data DA1 and the learned model PJ to determine the content information DC, thereby determining at least one of the display position PD, display time of image G, and display color or brightness of image G. Therefore, even when using the content information DC representing the same pattern, effects such as warnings or attention can be adjusted based on the content information DC.
[0083] Furthermore, as described above, the display system 1 also includes a second imaging device 22. Moreover, the processing device 33, based on the imaging results obtained by using the second imaging device 22 to photograph the first person H who has passed through the display position PD, obtains a first danger level DA based on first post-passage behavior data DA2, which represents the behavior of the first person H who has passed through the display position PD, and uses the first danger level DA to relearn the learned model PJ. Therefore, it is possible to reduce situations where the warning or attention effects of the image G based on content information DC are reduced due to human habits, etc.
[0084] Furthermore, as described above, the processing device 33 uses the display device 10 to display the first image G at the display position PD based on the first content information DC. Therefore, it is possible to obtain effects such as warnings or attention based on the image G with the content information DC.
[0085] Furthermore, as described above, the first content information DC used in the first image G contains image data. Therefore, the image G based on the content information DC can be displayed at the display position PD using the display device 10.
[0086] Furthermore, as described above, the display system 1 also includes a playback device 12 that plays audio around the display position PD. Additionally, the first content information DC used in the first image G includes sound data. Based on this, the processing device 33 uses the playback device 12 to play the sound based on the first content information DC. Therefore, effects such as warnings or attention based on the sound of the content information DC can be obtained. Thus, for example, even if the line of sight of a person H passing in front of the display position PD deviates from the display position PD, effects such as warnings or attention can still be obtained.
[0087] Furthermore, as described above, the first risk level DA is represented by the difference between a pre-defined feature quantity of the person H who has passed through the display position PD and the feature quantity shown in the first pass behavior data DA2. Therefore, it is possible to generate a learned model PJ or relearn the learned model PJ using the pass behavior data DA2.
[0088] Furthermore, as described above, the first pre-passage behavior data DA1 represents the characteristic quantity of the behavior of the first person H passing through the first region RE1. On the other hand, the first post-passage behavior data DA2 represents the characteristic quantity of the behavior of the first person H passing through the second region RE2, which is different from the first region RE1. Here, the display position PD is located between the first region RE1 and the second region RE2. Therefore, the pre-passage behavior data DA1 can be obtained based on the result of photographing the person H in the first region RE1. In addition, the post-passage behavior data DA2 can be obtained based on the result of photographing the person H in the second region RE2.
[0089] Furthermore, as described above, display system 1 uses program PG1. Program PG1 causes control device 30 to perform the following: based on the image capture result obtained by capturing a first person H before the display position PD of the first image G, which is based on the first content information DC, obtain first pre-passage behavior data DA1 representing the characteristic quantity of the behavior of the first person H before the display position PD; using a learned model PJ obtained by machine learning of the correspondence between the characteristic quantity of the behavior of the person H before the display position PD, the content information DC used in image G, and the danger level of the person H after the display position PD, and the first pre-passage behavior data DA1, determine the first content information DC to be used in the first image G. Control device 30 is an example of a "computer". Therefore, the above-described display system 1 can be implemented.
[0090] 2. Second Implementation Method
[0091] The second embodiment will now be described. Structures common to the first embodiment will be labeled with the same reference numerals, and their descriptions will be omitted. Furthermore, the following description will focus on aspects that differ from the first embodiment described above; identical aspects will be omitted.
[0092] Figure 6 This is a schematic diagram of the display system 1A according to the second embodiment. The display system 1A is configured the same as the display system 1 of the first embodiment, except that the installation location and the content displayed in the image G are different. Here, the display system 1A has an imaging device 20 instead of the first imaging device 21 and the second imaging device 22, and a control device 30A instead of the control device 30; otherwise, it is configured the same as the display system 1 of the first embodiment. The imaging device 20 is configured the same as the first imaging device 21 and the second imaging device 22, except that it captures both the first region RE1 and the second region RE2.
[0093] exist Figure 6 In the example shown, pathways AL1 and AL2 form a crossroads. Display system 1A uses image G to provide attentional alerts to person H_a traveling through pathway AL1. Here, the display position PD is located near the intersection of the crossroads. Furthermore, a first region RE1 is positioned behind the display position PD in the direction of person H_a's travel within pathway AL1. A second region RE2 is positioned in front of the display position PD in the direction of person H_a's travel within pathway AL1.
[0094] Furthermore, the width or shape of pathways AL1 and AL2 are not limited to Figure 6 The example shown can be arbitrary. Furthermore, the display position PD is not limited to... Figure 6The example shown could be located on the screen of passage AL1, on the left or right walls of passage AL1, or on the ceiling.
[0095] When person H_a in zone RE1 is about to encounter person H_b in passage AL2, control device 30A causes display device 10 to display image G. This serves as a warning to person H_a. Figure 6 The image G shown here illustrates the meaning of reminding person H_a to pay attention to the collision with person H_b.
[0096] According to the second embodiment described above, it is also possible to provide appropriate attention reminders to people H_a who are displayed at location PD.
[0097] 3. Third Implementation Method
[0098] The third embodiment will now be described. Structures common to the first embodiment will be labeled with the same reference numerals, and their descriptions will be omitted. Furthermore, the following description will focus on aspects that differ from the first embodiment described above; identical aspects will be omitted.
[0099] Figure 7 This is a schematic diagram of the display system 1B according to the third embodiment. The display system 1B is configured the same as the display system 1 of the first embodiment, except that the installation location and the content displayed in the image G are different. Here, the display system 1B has an imaging device 20 instead of the first imaging device 21 and the second imaging device 22, and a control device 30B instead of the control device 30; otherwise, it is configured the same as the display system 1 of the first embodiment. The imaging device 20 is configured the same as the first imaging device 21 and the second imaging device 22, except that it captures both the first region RE1 and the second region RE2.
[0100] exist Figure 7 In the example shown, display system 1B uses image G to draw attention to person H_a traveling through a path AL3 with a corner. Here, display position PD is located near the corner. Furthermore, a first region RE1 is positioned behind the display position PD in the direction of person H_a's travel within path AL3. A second region RE2 is positioned in front of the display position PD in the direction of person H_a's travel within path AL3.
[0101] Furthermore, the width or shape of the AL3 pathway is not limited to... Figure 7 The example shown can be arbitrary. Furthermore, the display position PD is not limited to... Figure 7 The example shown could be located on the screen of passage AL3, on the left or right walls of passage AL3, or on the ceiling.
[0102] When person H_a in zone RE1 is about to encounter person H_b in passage AL3, control device 30B causes display device 10 to display image G. This serves as a warning to person H_a. Figure 7 The image G illustrates the meaning of reminding person H_a, who is about to cross the center line of path AL3 and is turning at a corner, to travel to the right of person H_b.
[0103] According to the third embodiment described above, it is also possible to provide appropriate attention reminders to people H_a who are displayed at location PD.
[0104] 4. Fourth Implementation Method
[0105] The fourth embodiment will now be described. Structures common to the first embodiment will be labeled with the same reference numerals, and their descriptions will be omitted. Furthermore, the following description will focus on aspects that differ from the first embodiment described above; identical aspects will be omitted.
[0106] Figure 8 This is a block diagram of the display system 1C according to the fourth embodiment. The display system 1C uses program PG2 instead of program PG1 and uses the learned model PJC instead of the learned model PJ, otherwise it is configured the same as the display system 1 of the first embodiment described above.
[0107] The processing device 33 functions as an acquisition unit 33e, a decision unit 33f, a display control unit 33c, and a relearning unit 33g by executing program PG2.
[0108] The acquisition unit 33e is additionally equipped with the function of acquiring environmental data DE based on the first captured data DS1 or the second captured data DS2. Otherwise, it functions the same as the acquisition unit 33a in the first embodiment. The environmental data DE is data related to the surrounding environment of the display position PD, such as the presence of the step ST, the wetness or iciness of the ground FF, and darkness. The acquisition unit 33e extracts information related to the danger to person H as environmental data DE by performing image processing for object recognition from the first captured data DS1 or the second captured data DS2.
[0109] The decision unit 33f uses the prior behavioral data DA1 and the learned model PJC to determine the content information DC in a way that minimizes the risk to the person H who has passed the display location PD.
[0110] The learned model PJC is an estimation model obtained by machine learning the correspondence between the characteristics of the behavior of person H before passing through the display location PD, the content information DC used in the image G, and the environmental data DE in a way that minimizes the risk of person H passing through the display location PD.
[0111] The display control unit 33c causes the display device 10 to display an image G based on the content information DC determined by the decision unit 33f.
[0112] The relearning unit 33g uses the passed behavioral data DA2 to relearn the completed model PJC.
[0113] Figure 9 This is a flowchart illustrating the operation of the display system 1C according to the fourth embodiment. The operation of the display system 1C includes an additional step S13, and includes steps S4A, S9A, and S11A instead of steps S4, S9, and S11. Otherwise, the operation is the same as that of the display system 1 in the first embodiment.
[0114] In display system 1C, such as Figure 9 As shown, after step S3, in step S13, the acquisition unit 33e acquires environmental data DE. Next, in step S4A, the determination unit 33f determines content information DC using the preceding behavior data DA1 and the learned model PJC. Then, in step S5, the display control unit 33c causes the display device 10 to display an image G based on the content information DC.
[0115] Furthermore, after step S8, in step S9A, the relearning unit 33g stores the subsequent behavior data DA2 and its corresponding environmental data DE, the previous behavior data DA1, and the content information DC as history data in the storage device 32.
[0116] Furthermore, if it is determined in step S10 that relearning is needed (step S10: Yes), in step S11A, the relearning unit 33g performs relearning of the learned model PJC and then proceeds to step S12. On the other hand, if it is determined that relearning is not needed (step S10: No), the relearning unit 33g does not perform relearning of the learned model PJC and proceeds to step S12.
[0117] Figure 10 This diagram illustrates the learned-complete model PJC in the fourth embodiment. The learned-complete model PJC is an estimation model that outputs content information DC based on inputs of preceding action data DA1 and environment data DE. Specifically, the learned-complete model PJC is implemented by having the processing device 33 execute a program that generates content information DC based on the preceding action data DA1 and environment data DE, and a combination of multiple coefficients applied to this operation. This program is, for example, a program module constituting artificial intelligence software. These multiple coefficients are, for example, set using deep learning of the dataset DST1, described later. Figure 10In the example shown, the learned model PJC is a mathematical model consisting of a deep neural network with an input layer, an output layer, and intermediate layers. Furthermore, the number of intermediate layers is not limited to... Figure 10 The examples shown are arbitrary.
[0118] Figure 11 This is a diagram illustrating the machine learning process used to generate the learned model PJC in the fourth embodiment. The generation of the learned model PJC is as follows... Figure 11 As shown, machine learning is performed using a learning device with a learning processing unit 100C. Multiple datasets DST1 are used for this machine learning. Each dataset DST1 includes preceding behavior data DA1 and its corresponding hazard level DA, environmental data DE, and content information DC. The hazard level DA, environmental data DE, preceding behavior data DA1, and content information DC in dataset DST1 can also be generated using a device different from the control device 30C described above.
[0119] In dataset DST1, the risk level DA of person H after passing the display location PD, the content information DC at this time, the corresponding environmental data DE, and the group of people passing the previous behavior data DA1 are used.
[0120] The learning processing unit 100C sets multiple coefficients of the learned model PJC using taught machine learning on multiple datasets DST1. Specifically, the learning processing unit 100C updates the multiple coefficients of the learned model PJC to reduce the difference between the content information DC1 output by the temporarily learned model PJC based on the environmental data DE in dataset DST1 and the content information DC with the lowest risk DA contained in dataset DST1. For example, the learning processing unit 100C repeatedly updates the multiple coefficients of the learned model PJC using the error backpropagation method to minimize the evaluation function representing the difference between content information DC and content information DC1. Based on the potential trends between the environmental data DE, the preceding data DA1, and the content information DC in multiple datasets DST1, the learned model PJC outputs statistically appropriate content information DC for unknown environmental data DE and preceding data DA1. Thus, it is possible to change the image data of the content information DC according to the environmental data DE and the preceding data DA1, or to change the display method such as the position, time, brightness, and hue of the image G displayed based on the image data. The environmental data DE used in the output of the content information DC is equivalent to the first environmental data.
[0121] According to the fourth embodiment described above, it is also possible to provide appropriate attention reminders to a person H passing through the display location PD. In this embodiment, as described above, the correspondence of the learned model PJC includes the relationship between the preceding behavior data DA1, the content information DC, the hazard level DA, and the environmental data DE related to the surrounding environment of the display location PD. The processing device 33 obtains the first environmental data DE when the first person H passes through the display location PD, and uses the first environmental data DE, the first preceding behavior data DA1, and the learned model PJC to determine the first content information DC. Therefore, it is possible to determine the appropriate content information DC based on changes in the surrounding environment.
[0122] Furthermore, as mentioned above, the environment surrounding the display location PD is an environment related to the danger to the person H in the area containing the display location PD. Therefore, information related to the environment surrounding the display location PD can be obtained based on the results of photographing that area.
[0123] 5. Fifth Implementation Method
[0124] The fifth embodiment will now be described. Structures common to the first embodiment will be labeled with the same reference numerals, and their descriptions will be omitted. Furthermore, the following description will focus on aspects that differ from the first embodiment described above; identical aspects will be omitted.
[0125] Figure 12 This is a diagram illustrating an example of the score for each action based on the preceding action data DA1. It is also possible to classify the actions that may occur before the display position PD, and generate a learned model PJ using the preceding action data DA1 corresponding to each action, the corresponding following action data DA2, and the content information DC. Thus, it is possible to... Figure 12 Each behavior a, b, c, d, e in the model uses the best learned model PJ to select the content information DC that represents the warning to be displayed.
[0126] Figure 12 The behaviors categorized are a, b, c, d, e... for example, "speeding," "tripping," "walking while using a phone," "hands in pockets," "caution against collisions," etc. Furthermore, the number of behaviors categorized is not limited to... Figure 12 The examples shown are arbitrary.
[0127] 6. Variations
[0128] The methods illustrated above can be modified in various ways. The following examples illustrate specific modifications applicable to the methods described above. Any two or more methods selected from the following examples can be appropriately combined without contradiction.
[0129] 6-1. Variation Example 1
[0130] In the above embodiments, a structure in which the control device 30 and the display device 10 are separate is illustrated, but the structure is not limited to this; the control device 30 may also be part of the display device 10.
[0131] 6-2. Variation Example 2
[0132] Among the above embodiments, a structure using a projection device in a display device is illustrated, but the display device is not limited to this structure. The display device may also use a device that uses a liquid crystal display panel, an organic EL (electro-luminescence) panel, or the like as the display surface as the optical device.
[0133] 7. Summary of this disclosure
[0134] The following is a summary published in this note.
[0135] (Appendix 1) The display system includes: a display device that displays an image based on content information at a display location; a first capturing device; a storage device that stores a learned model obtained by machine learning of a correspondence between pre-passage behavior data representing feature quantities of a person's behavior before passing the display location (captured by the first capturing device), content information used in the image, and the danger level of the person who passed the display location; and a processing device that processes the data as follows: based on a capture result obtained by capturing a first person before passing the display location using the first capturing device, it obtains first pre-passage behavior data representing feature quantities of the first person's behavior before passing the display location; and uses the first pre-passage behavior data and the learned model to determine first content information to be used in a first image corresponding to the first pre-passage behavior data. Thus, by using the pre-passage behavior data and the learned model to determine the content information of the image displayed by the display device, the danger level of the person who passed the display location is minimized, thereby effectively drawing attention through the image. Therefore, by using the learned model, the corresponding first content information can be determined based on the first pre-passage behavior data. As a result, it is possible to provide appropriate attention reminders to people passing by the displayed location.
[0136] (Appendix 2) According to the display system in Appendix 1, the correspondence includes the relationship between the preceding behavior data, the content information, the danger level, and environmental data regarding the surrounding environment of the display location. The processing device performs the following processing: obtaining first environmental data when the first person passes through the display location; using the first environmental data, the first preceding behavior data, and the learned model to determine the first content information. Thus, appropriate content information can be determined based on changes in the surrounding environment.
[0137] (Appendix 3) According to the display system described in Appendix 1 or 2, the processing device performs the following processing: using the first image obtained from previous data and the learned model, it determines at least one of the following: the display position, the display time of the first image, the display color of the first image, and the brightness of the first image. Thus, even when using content information representing the same pattern, it is possible to adjust the warning or attention-grabbing effects of the image based on the content information.
[0138] (Appendix 4) According to any one of Appendices 1 to 3, the display system further includes a second imaging device, and the processing device performs the following processing: obtaining a first danger level based on the imaging result obtained by using the second imaging device to photograph the first person passing through the display location, the first danger level being based on first post-passage behavior data representing a feature quantity of the behavior of the first person passing through the display location; and using the first danger level, re-performing the learned model. This reduces situations where warnings or attention effects based on content-information images are diminished due to human habits, etc.
[0139] (Note 5) According to any one of the display systems in Notes 1 to 4, the processing device uses the display device to display the first image at the display position based on the first content information. This results in effects such as warnings or alerts based on the content information-based image.
[0140] (Note 6) According to any one of the display systems in Notes 1 to 5, the first content information used in the first image includes image data. Therefore, it is possible to display an image based on the content information at a display location using a display device.
[0141] (Appendix 7) According to the display system described in Appendix 6, the display system further includes a playback device that plays audio around the display position. The first content information used in the first image includes sound data, and the processing device performs the following processing: plays audio based on the first content information using the playback device. This results in effects such as warnings or attracting attention based on the audio content information. Therefore, for example, even if a person's line of sight deviates from the display position before it, effects such as warnings or attracting attention can still be obtained.
[0142] (Note 8) According to the display system described in Note 4, the first degree of danger is represented by the difference between a pre-defined feature quantity of the person who has passed through the display location and the feature quantity represented by the first post-passage behavior data. Therefore, the post-passage behavior data can be used to create a learned model or for relearning.
[0143] (Appendix 9) According to the display system described in Appendix 4 or 8, the first method uses forward behavior data to represent the characteristic quantity of the behavior of the first person traveling in the first area, and the first method uses subsequent behavior data to represent the characteristic quantity of the behavior of the first person traveling in the second area, which is different from the first area. The display position is located between the first area and the second area. Thus, forward behavior data can be obtained based on the result of photographing a person in the first area. Similarly, subsequent behavior data can be obtained based on the result of photographing a person in the second area.
[0144] (Note 10) According to the display system described in Note 2, the surrounding environment of the display location is an environment related to the danger to the person in the area containing the display location. Therefore, based on the results of photographing the area, information related to the surrounding environment of the display location can be obtained.
[0145] (Appendix 11) A display device that displays an image based on content information at a display location, the display device comprising: a storage device storing a learned model obtained by machine learning of a correspondence between pre-passage behavior data representing feature quantities of a person's behavior before passing the display location, content information used in the image, and the danger level of a person who has passed the display location; and a processing device that performs the following processing: obtaining first pre-passage behavior data representing feature quantities of the behavior of the first person before passing the display location based on an image taken of a first person before passing the display location; and determining, using the first pre-passage behavior data and the learned model, first content information to be used in a first image corresponding to the first pre-passage behavior data. Thus, appropriate attention reminders can be given to a person passing the display location.
[0146] (Appendix 12) A recording medium containing a program that causes a computer to perform the following processing: obtaining first pre-passage behavior data representing feature quantities of the behavior of the first person before passing the display position of a first image based on first content information, based on a photographic result obtained by photographing a first person before passing the display position; and determining first content information used in the first image using a learned model obtained by machine learning of the correspondence between the feature quantities of the behavior of the person before passing the display position, the content information used in the image, and the danger level of the person after passing the display position, and the first pre-passage behavior data. Thus, appropriate attentional reminders can be given to the person passing the display position.
Claims
1. A display system having: A display device that displays an image based on content information at a display location; First camera unit; A storage device storing a learned model, which is obtained by machine learning of the correspondence between pre-passage behavior data representing feature quantities of a person's behavior before passing the display location (captured by the first imaging device), content information used in the image, and the danger level of the person who has passed the display location; and Processing device, The processing device performs the following processing: Based on the imaging results obtained by using the first imaging device to capture images of the first person before passing the display position, first pre-passage behavior data representing characteristic quantities of the behavior of the first person before passing the display position is obtained; and Using the first preceding data and the learned model, determine the first content information to be used in the first image corresponding to the first preceding data. The correspondence includes the relationship between the preceding behavior data, the content information, the danger level, and environmental data regarding the surrounding environment of the display location. The environmental data includes at least one of the following: (1) Are there steps in front of the displayed position? (2) Whether the ground in front of the display location is wet or icy; (3) Whether the area in front of the display position is dark.
2. The display system according to claim 1, wherein, The processing device performs the following processing: Obtain first environmental data when the first person passes through the displayed location; and The first content information is determined using the first environmental data, the first preceding behavior data, and the learned model.
3. The display system according to claim 1, wherein, The processing device performs the following processing: The first method uses the preceding behavior data and the learned model to determine at least one of the following: the display position, the display time of the first image, the display color of the first image, and the brightness of the first image.
4. The display system according to claim 1, wherein, The display system also has a second shooting device. The processing device performs the following processing: A first danger level is obtained based on the shooting results obtained by using the second shooting device to shoot the first person passing through the display position. The first danger level is based on first post-passage behavior data, which is a feature quantity representing the behavior of the first person passing through the display position. Using the first level of risk, relearning is performed on the learned model.
5. The display system according to claim 1, wherein, The processing device uses the display device to display the first image at the display position according to the first content information.
6. The display system according to claim 1, wherein, The first content information used in the first image includes image data.
7. The display system according to claim 6, wherein, The display system also includes a playback device that plays music around the display location. The first content information used in the first image includes audio data. The processing device performs the following processing: The playback device is used to play sound based on the first content information.
8. The display system according to claim 4, wherein, The first level of danger is represented by the difference between a pre-defined characteristic quantity of the person after passing the displayed location and the characteristic quantity represented by the first pass behavior data.
9. The display system according to claim 4, wherein, The first characteristic quantity, represented by the preceding behavior data, indicates the behavior of the first person traversing the first area. The first post-travel behavior data represents the characteristic quantity of the first person's behavior when traversing a second area different from the first area. The display position is located between the first region and the second region.
10. A display device for displaying an image based on content information at a display position, wherein, The display device has: A storage device storing a learned model, which is obtained by machine learning of pre-passage behavior data representing feature quantities of a person's behavior before passing the display location, content information used in the image, and the correspondence between the risk level of the person who has passed the display location; and Processing device, The processing device performs the following processing: Based on the photographic results obtained from photographing the first person before they pass the displayed position, first pre-passage behavior data representing feature quantities of the first person's behavior before they pass the displayed position is obtained; and Using the first preceding data and the learned model, determine the first content information to be used in the first image corresponding to the first preceding data. The correspondence includes the relationship between the preceding behavior data, the content information, the danger level, and environmental data regarding the surrounding environment of the display location. The environmental data includes at least one of the following: (1) Are there steps in front of the displayed position? (2) Whether the ground in front of the display location is wet or icy; (3) Whether the area in front of the display position is dark.
11. A recording medium containing a program that causes a computer to perform the following processes: Based on the image capture results obtained from photographing a person before their display location in an image based on content information, pre-passage behavior data is obtained, representing a feature quantity indicating the person's behavior before passing the display location; and Using a learned model derived from machine learning, which establishes a correlation between features of a person's behavior before passing the displayed location, content information used in the image, and the perceived danger level of the person after passing the displayed location, and the pre-passage behavior data, the content information to be used in the image is determined. The correspondence includes the relationship between the preceding behavior data, the content information, the danger level, and environmental data regarding the surrounding environment of the display location. The environmental data includes at least one of the following: (1) Are there steps in front of the displayed position? (2) Whether the ground in front of the display location is wet or icy; (3) Whether the area in front of the display position is dark.
Citation Information
Patent Citations
Display control system and recording medium
JP2014123277A
Alarm pre-control method for preventing personal injury
CN111063161A
Driver danger early warning method, device and equipment and storage medium
CN114360291A
