Systems and methods for displaying video data in a target environment

Abstract

A system for automatically displaying output information on a display surface based on activities in a target environment, including an image capture device for capturing target environment data in real time, a processing unit for processing the data to determine the position for displaying the output information and determine the content of the output information, and a video output unit for displaying the output information on the display surface. The video output unit is configured to rotate in a 360° manner, wherein the rotation angle of the video output unit is controlled by the processing unit. The image capture device also captures in real time the output information displayed on the display surface and provides feedback data to the processing unit in real time, so that the processing unit can control the focus, direction, and the output information displayed on the video output unit.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for displaying video data in a target environment, and more particularly to displaying video data using real-time interactive visual and audio cues based on a closed-loop artificial intelligence (AI)-driven process. Background Art

[0002] Companies typically spend a large amount of money advertising newly launched products on display screens in public places such as shopping malls, railway stations / bus stops, etc. To maximize profit and return on investment, companies desire the largest number of customers to purchase the new product. However, existing display screens used for advertising only display pre-programmed data. In particular, existing display screens do not display advertisements based on the target customers.

[0003] In addition, in a self-checkout (SCO) store, the cashier does not physically appear at the billing counter to process billing for the customer. Therefore, in an SCO store, a system is needed that provides an audio / video alarm based on the observation of theft behavior. Similarly, in the case where a customer cannot correctly use the SCO system, a system is needed to alert the store supervisor that the customer may require attention and assistance.

[0004] In addition, in the event of a natural disaster (such as an earthquake, fire, tsunami, etc.), it may be necessary to issue an alarm to alert people in public places of the disaster and provide them with directions to a safer place.

[0005] In addition, current systems for guiding / training students fail to observe the student's environment and adjust the guidance / training accordingly. For example, current training systems execute pre-established procedures for training medical or veterinary students and do not take into account the student's environment.

[0006] Therefore, in view of the above, there is a need for a system that takes into account the environment of one or more target users and provides automated audio / video output accordingly. Summary of the Invention

[0007] In one aspect of the present disclosure, a system for automatically displaying output information on a display surface based on one or more activities in a target environment is provided. The system includes an image capture device configured to capture images and video data of the target environment in real time to identify one or more activities. The system may include a processing unit configured to process the images and the video data to determine a location for displaying the output information and to determine the content of the output information. The system may further include a video output unit configured to display the output information on the display surface, wherein the video output unit is configured to rotate in a 360° manner, and wherein the rotation angle of the video output unit is controlled by the processing unit. The image capture device is further configured to capture in real time the output information displayed on the display surface and to provide feedback data to the processing unit in real time so that the processing unit can control the focus, direction, and the output information displayed on the video output unit.

[0008] In another aspect of the present disclosure, a method for automatically displaying output information on a display surface based on one or more activities in a target environment is provided. The method includes capturing images and video data of the target environment in real time to identify one or more activities. The method may further include processing the images and video data to determine a location for displaying the output information and to determine the content of the output information. The method may further include displaying the output information on the display surface through a video output unit, wherein the video output unit is configured to rotate in a 360° manner. The method may further include controlling the rotation angle of the video output unit based on the captured images and video data. The method may further include capturing in real time the output information displayed on the display surface and generating feedback data to control the focus, direction, and the output information displayed on the video output unit.

[0009] In yet another aspect of the present disclosure, there is provided a computer-programmable product for automatically displaying output information on a display surface based on one or more activities in a target environment. The computer-programmable product includes a set of instructions that, when executed by a processor, cause the processor to: capture images and video data of the target environment in real time to identify one or more activities; process the images and video data to determine a location for displaying the output information and to determine the content of the output information; generate the output information based on the processed data; and display the output information on the display surface via a video output unit. The video output unit is configured to rotate in a 360° manner and to control the rotation angle of the video output unit based on the captured images and video data. Additionally, the output information displayed on the display surface is captured in real time and feedback data is generated to control the focus, orientation of the video output unit, and the output information displayed on the video output unit.

[0010] Various embodiments of the present disclosure provide a system for capturing human behavior and interacting and communicating with humans to guide or inform humans in a manner suitable for the observed process and environment. The system can receive visual, audio, and other sensor inputs and create visual and audio outputs to form a closed-loop interaction managed by software intelligence running in the background. The system can further act as an intelligent instructor / coach / supervisor, allowing for automatic ensuring of optimal performance in meeting standards and prescribed processes. The system provides an opportunity for two-way communication with the user, using a camera for input, a projector for output, and being controlled by AI software.

[0011] The system is useful in scenarios where users need to be guided or trained. The camera can observe the environmental process and the use of the projector, guide the user to take actions according to the expected results, and keep training records. Another example is training medical or veterinary students to perform specific procedures. The closed-loop feedback ensures that the AI software is in real-time control and can change or correct processes and activities as they occur. The closed-loop AI-driven process control uses real-time interactive visual and audio prompts / nudges to guide, control, and / or ensure optimal process or behavioral outcomes.

[0012] It should be understood that the features of the present disclosure are readily combinable in various combinations without departing from the scope of the present disclosure as defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above - described invention content and the subsequent detailed description of exemplary embodiments can be better understood when read in conjunction with the accompanying drawings. To illustrate the present disclosure, exemplary configurations of the present disclosure are shown in the drawings. However, the present disclosure is not limited to the specific methods and means disclosed herein. In addition, those skilled in the art will understand that the drawings are not drawn to scale. Wherever possible, like elements are denoted by the same reference numerals.

[0014] Figure 1 is a schematic diagram of an example system for displaying video data based on activities in a target environment according to an embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram of an example operation for processing image frames captured by an image - capture device according to an embodiment of the present disclosure;

[0016] Figure 3A is a schematic diagram of an example video output unit for projecting predefined image / video data generated by a processing unit onto a display surface according to an embodiment of the present disclosure;

[0017] Figure 3B is a schematic diagram of an example mechanism for rotating an electric mirror about a vertical axis and a horizontal axis in a mirror plane according to an embodiment of the present disclosure;

[0018] Figure 3C shows an example motor including two electrically - controlled rods according to an embodiment of the present disclosure; and

[0019] Figure 4 is an example flowchart showing a method for automatically displaying video data on a display surface based on one or more activities in a target environment.

[0020] In the drawings, underlined numbers are used to represent the item above the underlined number or the item adjacent to the underlined number. Non - underlined numbers are related to the item identified by the line connecting the non - underlined number to a certain item. When a non - underlined number is accompanied by a related arrow, the non - underlined number is used to identify the general item pointed to by the arrow Detailed Description

[0021] The following detailed description shows embodiments of the present disclosure and how they can be implemented. Although the best mode for carrying out the present disclosure has been disclosed, those skilled in the art will recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0022] Figure 1FIG. 0 is a schematic diagram of an example system 100 for displaying video data based on activities in a target environment 101 according to an embodiment of the present disclosure. In one embodiment of the present disclosure, the target environment 101 may belong to a self-checkout (SCO) store, and although not shown, may include entities such as products, conveyor belts, industrial robots, and activities such as an operator entering or leaving the scene; picking up, putting down, moving, weighing, or scanning items; operating a touch display screen; and activities such as paying by cash, mobile electronic transaction, or credit card. However, it will be apparent to those of ordinary skill in the art that the target environment 101 may belong to a general industrial area, a training hall, a security inspection area, a shopping mall, and a restaurant. In addition to the target environment 101 specified above, the system 100 may be used for retail automation, customer upselling, employee guidance, employee training, logistics automation (in and out of goods), medical guidance, such as EG surgical training, surgical expert training with visual cues, emergency instructions in the event of a fire or earthquake.

[0023] The system 100 includes a processing unit 102, an image capture device 104, an audio recording device 106, a sensor unit 108, and a video output unit 110, which are communicatively connected to each other through a communication network. The communication network may be any suitable wired network, wireless network, combination of wired and wireless networks, or any other conventional network without limiting the scope of the present disclosure. Some examples may include a local area network (LAN), a wireless LAN connection, an Internet connection, a point-to-point connection, or other network connections and combinations thereof. In one example, the network may include a mobile communication network, such as a 2G, 3G, 4G, or 5G mobile communication network. The communication network may be connected to one or more other networks to provide connections between a larger number of devices. For example, this may be the case when the networks are connected together through the Internet.

[0024] The image capture device 104 is configured to capture one or more images and videos of the target environment 101 in real time to identify the actions of various entities (such as people, animals, and things) in the image frames, and transmit the captured data to the processing unit 102. Examples of the image capture device 104 include, but are not limited to, 360° cameras, closed-circuit television (CCTV) cameras, high-definition (HD) cameras, non-HD cameras, hand-held cameras, traffic cameras, police car cameras, and cameras on unmanned aerial vehicles (UAVs).

[0025] The audio recording device 106 is configured to record audio data from the target environment 101 and transmit the recorded audio data to the processing unit 102. Examples of the audio recording device 106 include, but are not limited to, dynamic microphones, capacitive microphones, piezoelectric microphones, or ribbon microphones.

[0026] The sensor unit 108 is configured to detect events or changes in the target environment 101 and transmit the detected information to the processing unit 102. In one embodiment, the target environment 101 is one or more residential / commercial buildings, and the sensor unit 108 may include sensors installed therein to detect the occurrence of natural disasters inside or near the building; and transmit the detected information to the processing unit 102. In yet another embodiment, the target environment 101 is a security checkpoint area at an airport, a shopping (or other) mall, or other similar locations; and the sensor unit 108 may include installed X-ray equipment to detect the presence of suspicious items in the luggage, bags, clothes, or other aspects of the people on-site; and transmit the detected information to the processing unit 102.

[0027] The processing unit 102 is a central control unit that controls the operations of the image capture device 104, the audio recording device 106, the sensor unit 108, and the video output unit 110. The processing unit 102 is configured to process the input data received from the image capture device 104, the audio recording device 106, and the sensor unit 108, generate output information based on the processed data, and display the output information on the display surface 112 through the video output unit 110. In one embodiment of the present disclosure, the output information may include predefined video and audio outputs, including but not limited to alarms, notifications, advertisements, instructions, and training videos of the target environment 101. In one embodiment of the present disclosure, the video output unit 110 may include a projector; and the display surface 112 may include a white projection screen, a gray projection screen, or a white wall for displaying the projection image from the projector. In one embodiment of the present disclosure, the processing unit 102 is configured to control the focus and direction of the video output unit 110; and also control the content displayed through the video output unit 110. In another embodiment of the present disclosure, the video output unit 110 may include a non-projection display, such as a light-emitting diode (LED), a liquid crystal display (LCD), and an organic light-emitting diode (OLED).

[0028] In one embodiment of the present disclosure, the image capture device 104 captures the image / video data displayed on the display surface 112, and the audio recording device 106 records the audio data being played on the display surface 112. The processing unit 102 uses the feedback data to improve the quality of the output information displayed on the display surface 112 to enhance user feedback.

[0029] In one embodiment of the present disclosure, the processing unit 102 includes an artificial intelligence (AI) platform that can direct visual information to an exact location in the target environment 101. The processing unit 102 can be implemented locally on a local computing device or a remote processing server. In the context of the present disclosure, the processing unit 102 can include an AI-based processor, a graphics processing unit (GPU) for processing video / image data, and a memory for storing one or more instructions. Throughout the present disclosure, the term "AI-based processor" refers to a computing element that can be used to respond to and process instructions stored in a corresponding memory. Optionally, the AI-based processor includes, but is not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or any other type of processing circuit. The processing unit 102 can include a set of initial predefined outputs that are programmed and enhanced by dynamic outputs created by corresponding AI software.

[0030] Figure 2 is a schematic diagram of an example operation for processing an image frame 202 captured by an image capture device 104 (such as Figure 1 shown). The image frame 202 is circular and has been captured by a 360° lens of the image capture device 104. The processing unit 102 (such as Figure 1 shown) receives the image frame 202 and creates a flattened representation of the image frame 202 to generate a flattened image 204. In one embodiment of the present disclosure, the processing unit 102 (such as Figure 1 shown) compensates for fisheye distortion while creating the flattened image 204 and generates a grid view image 206 of the flattened image 204.

[0031] The grid view image 206 includes an active grid referenced to the target environment 101 (such as Figure 1 shown). In one implementation, the grid characteristics (i.e., the number of rows and columns) can be configured by the user. In another implementation, the grid characteristics are dynamically variable. In this case, the grid characteristics are determined by the processing unit 102, which analyzes the motion detected in the view of the image capture device 104 and ensures that the grid reference can be used to distinguish the positions of each individual element in the scene. Then, the grid reference is used to notify the processing unit 102 of the activities occurring in the target environment 101 and the location of the activities.

[0032] In one example, when the target environment 101 is a shopping mall, the image 204 can show locations that are usually crowded with people. The processing unit 102 can use this information to display a predefined video advertisement at the corresponding location to focus the attention of the maximum number of people on the advertisement. In another example, when the target environment 101 is a security check area at an airport, a shopping (or other) mall, or other similar locations, the image 204 can be an X-ray image showing one or more suspicious prohibited items or articles present. The processing unit 102 can use this information to generate an audio / video alert for the relevant security supervisor. Reference has been made to Figure 3A explained in detail the projection of video / image data onto a display surface.

[0033] Figure 3A is a schematic diagram of an example video output unit 300 (similar to the video output unit 110) for projecting predefined image / video data generated by a processing unit 301 (similar to the processing unit 102) onto a display surface 310 (similar to the display surface 112) according to an embodiment of the present disclosure.

[0034] The video output unit 300 includes a light source 302, a lens 304, an electric focusing system 306, and an electric mirror 308. Examples of the light source 302 can include, but are not limited to, lasers, lamps, or LEDs (light emitting diodes). The lens 304 can include a lens with optical and digital zoom and focusing capabilities. The electric focusing system 306 is configured to direct the light from the light source 302, reflected by the lens 304, to the electric mirror 308. The electric mirror 308 is configured to move on one or each of a vertical axis and a horizontal axis to project a predefined image / video onto the display surface 310 at different inclinations / positions. In an embodiment of the present disclosure, the electric mirror 308 is configured to rotate a predefined image / video by up to 90° to display a predefined image at a predefined position on the display surface 310. Reference Figure 3B explains in detail the mechanism for rotating the electric mirror 308.

[0035] Figure 3B Shows a mechanism 312 for rotating the electric mirror 308 about a vertical axis and a horizontal axis in the mirror plane. The mechanism 312 includes a motor 314 fixed to the ceiling and connected to the electric mirror 308 and a physical hook 315 to ensure that the movement rod 317 remains in a horizontal position. The electric mirror 308 is fixedly connected to the ceiling. Therefore, the normal axis of the electric mirror 308 is fixed, thereby restricting its movement along the depth axis.

[0036] Figure 3C Shows that the motor 314 can include two electrically controlled rods 316a and 316b for controlling the rotation of the electric mirror 308 in the horizontal and vertical directions.

[0037] Return reference Figure 3B , the motor 314 is controlled in real time by the processing unit 301 (similar to the processing unit 102), such that when the corresponding sensor unit transfers an action to the processing unit 318, the processing unit 301 generates a relevant video / image output and controls the motor 314 to direct the video / image output to an exact position on the display surface 310. In the context of the present disclosure, the electric mirror 308 is configured to rotate in a 360° manner according to the instructions of the processing unit 318.

[0038] Return reference Figure 1 , the image capture device 104 is configured to capture video / image data displayed on the output surface 112 and provide it as feedback data to the processing unit 102. For example, if the projection is not clear enough, the processing unit 102 can adjust the focusing lens of the video output unit 110 to improve the quality of the output image / video data.

[0039] In one embodiment of the present disclosure, based on previous experience and software learning, in order to avoid and prevent such undesired activities, the processing unit 102 is configured to predict when an undesired activity may occur. Additionally, the processing unit 102 can build a risk distribution for use in "hypothetical" scenario analysis. For example, the processing unit 102 can detect that the process is not being followed correctly and highlight it to the user. However, the processing unit 102 can view this scenario as both a potential negative risk and a potential positive risk at the same time. The processing unit 102 can predict the expected outcomes of both situations, and if it sees an unexpected benefit emerging from the process deviation, it can automatically change the process to incorporate the change into an improved process. This can occur at an independent site in order to observe the results before pushing the change to the entire system. The processing unit 102 can store the previous process in case it is necessary to roll back to the original state due to some unforeseen reason.

[0040] Figure 4 is an example flowchart showing a method 400 for automatically displaying video data on a display surface based on one or more activities in a target environment according to the present disclosure. The method is described as a collection of steps in a logic flowchart that represents a series of steps that can be implemented using hardware, software, or a combination thereof.

[0041] In step 402, images and video data of the target environment are captured in real time to identify one or more activities occurring therein. At least one of the following can be used to capture the images and video data: a 360° camera, a closed-circuit television (CCTV) camera, a high-definition (HD) camera, a non-HD camera, a handheld camera, a traffic camera, a police car camera, and a camera on an unmanned aerial vehicle (UAV).

[0042] In step 404, the image and video data are processed, and output information is generated based on the processed data. According to an embodiment of the present disclosure, a graphics processing unit (GPU) is used to process the video / image data. In an embodiment of the present disclosure, the output information includes predefined video and audio data, including at least one of the following: alerts, notifications, advertisements, instructions, and training videos. Additionally, according to an embodiment of the present disclosure, the output information is generated based on a raster view image. The raster view image is generated by receiving a circular input image frame and creating a flattened representation of the circular input image frame to generate the raster view image of the flattened representation. Further, the raster view image provides information about the position of one or more entities in the circular input image frame.

[0043] In step 406, the output information is displayed on a display surface by a video output unit. In an embodiment of the present disclosure, the video output unit is configured to rotate in a 360° manner. In an embodiment of the present disclosure, the video output unit includes a projector, and the display surface includes at least one of the following: a white projection screen for displaying a projected image, a gray projection screen, and a white wall.

[0044] In step 408, the rotation angle of the video output unit is controlled based on the captured image and video data. In an embodiment of the present disclosure, the above-mentioned video output unit includes a light source, a lens, an electric mirror, and an electric focusing system. The electric mirror is configured to move in the horizontal and vertical directions to project a predefined image onto one or more positions on the display surface, and the movement of the electric mirror is controlled by one or more electric control rods based on the rotation angle. Additionally, the electric focusing system is configured to direct the light from the light source, reflected by the lens, to the electric mirror.

[0045] In step 410, the captured output information is displayed on the display surface in real time, and feedback data is generated to control the focus and direction of the video output unit; and further control the output information displayed on the video output unit. In one example, if the projection is not clear enough, the focusing lens of the video output unit is adjusted to improve the display quality of the output information. This is achieved by establishing a feedback loop between the projected image and the camera input. The AI software may detect the out-of-focus image and send a correction signal to the lens to improve the image quality. This is an ongoing cycle because the display surface may move or change shape.

[0046] Without departing from the scope of the present disclosure as defined by the appended claims, modifications may be made to the embodiments of the present disclosure described above. Expressions such as "including", "comprising", "incorporating", "consisting of", "having", "being" are intended to be interpreted in a non-exclusive manner, i.e., allowing the existence of items, components, or elements that are not explicitly described. References to the singular should also be interpreted as relating to the plural.

Claims

1. A system for automatically displaying output information on a display surface based on the positions and activities of one or more entities in a target environment, the system comprising: An image capture device configured to capture images and video data of the target environment using a fish-eye view to generate a circular input image frame of the target environment; A processing unit configured to: Receive from the image capture device the circular input image frame representing the fish-eye view of the target environment; Generate an image frame having a flattened representation of the target environment based on the circular input image frame; Generate a grid view image of the image frame having the flattened representation, wherein the grid view image includes a real-time grid reference of the target environment on the image frame having the flattened representation, and the real-time grid reference is used by the processing unit to determine information about the positions and the activities of the one or more entities in the target environment; And A video output unit configured to display output information on the display surface based on information determined according to the real-time grid reference, wherein the video output unit is configured to rotate in a 360° manner, and wherein the rotation angle of the video output unit is controlled by the processing unit, Wherein the image capture device is further configured to capture in real time the output information displayed on the display surface and provide feedback data to the processing unit in real time, so that the processing unit can control the focus, direction and content of the information displayed on the display surface by the video output unit based on the feedback data.

2. The system according to claim 1, further comprising an audio recording device configured to record audio data of the target environment and transmit the recorded audio data to the processing unit, wherein the audio recording device is configured to record audio data of the output information as feedback data and provide the feedback data to the processing unit, so that the processing unit can control the focus, the direction and the content of the information displayed on the display surface by the video output unit in real time based on the feedback data of the audio recording device.

3. The system according to claim 1, wherein the processing unit includes an artificial intelligence (AI) platform configured to direct visual information to one or more predefined positions in the target environment.

4. The system according to claim 1, wherein the processing unit includes a graphics processing unit (GPU) for processing video / image data.

5. The system according to claim 1, wherein the output information includes predefined video and audio data, including at least one of the following: alarms, notifications, advertisements, instructions and training videos.

6. The system according to claim 1, wherein the video output unit includes a projector, and the display surface includes at least one of the following: a white projection screen for displaying a projected image, a gray projection screen and a white wall.

7. The system according to claim 1, wherein the processing unit is configured to generate and display the output information based on the grid view image.

8. The system according to claim 1, further comprising a sensor unit configured to detect one or more events and changes in the target environment, wherein the sensor unit includes at least one of the following: radar, X-ray equipment, a sensor for detecting the occurrence of natural disasters.

9. A method for automatically displaying output information on a display surface based on the positions and activities of one or more entities in a target environment, the method comprising: Capturing an image and video data of the target environment using a fish-eye view of an image capture device to generate a circular input image frame of the target environment; Receiving, from the image capture device, the circular input image frame representing the fish-eye view of the target environment; Generating an image frame having a flattened representation of the target environment based on the circular input image frame; Generating a grid view image of the image frame having the flattened representation, wherein the grid view image includes a real-time grid reference of the target environment on the image frame having the flattened representation, and the real-time grid reference is used to determine information about the positions and the activities of the one or more entities in the target environment; and Displaying output information on the display surface by a video output unit based on the information determined according to the real-time grid reference, wherein the video output unit is configured to rotate in a 360° manner; Controlling the rotation angle of the video output unit based on the captured image and video data; and Capturing in real time the output information displayed on the display surface and generating feedback data to control the focus, direction, and content of the information displayed on the display surface by the video output unit based on the feedback data.

10. The method according to claim 9, further comprising: Recording audio data of the target environment and transmitting the recorded audio data; And Recording audio data of the output information as feedback data and providing the feedback data for real-time control of the focus, the direction, and the content of the information displayed on the display surface by the video output unit based on the feedback data.

11. The method according to claim 9, further comprising using an artificial intelligence (AI) platform to direct visual information to one or more predefined locations in the target environment.

12. The method according to claim 9, further comprising processing the video / image data using a graphics processing unit (GPU).

13. The method according to claim 9, wherein the output information includes predefined video and audio data, including at least one of the following: alarms, notifications, advertisements, instructions, and training videos.

14. The method according to claim 9, wherein the video output unit includes a projector, and the display surface includes at least one of the following: a white projection screen, a gray projection screen, and a white wall for displaying a projection image.

15. The method according to claim 9 further comprises: Generating and displaying the output information based on the grid view image.

16. The method according to claim 9 further comprises detecting one or more events and changes in the target environment using sensors selected from the following group: radar, X-ray equipment, sensors for detecting the occurrence of natural disasters.

17. A computer-programmable product for automatically displaying output information on a display surface based on the position and activity of one or more entities in a target environment, the computer-programmable product comprising a set of instructions which, when executed by a processor, cause the processor to: Capture images and video data of the target environment using a fisheye view of an image capture device to generate a circular input image frame of the target environment; Receive from the image capture device the circular input image frame representing the fisheye view of the target environment; Generate an image frame having a flattened representation of the target environment based on the circular input image frame; Generate a grid view image of the image frame having the flattened representation, wherein the grid view image includes a real-time grid reference of the target environment on the image frame having the flattened representation, and the real-time grid reference is used to determine information about the position and the activity of the one or more entities in the target environment; and Display output information on the display surface by a video output unit based on information determined according to the real-time grid reference, wherein the video output unit is configured to rotate in a 360° manner; Control the rotation angle of the video output unit based on the captured images and video data; and Capture in real time the output information displayed on the display surface and generate feedback data to control the focus, direction, and content of the information displayed on the display surface by the video output unit based on the feedback data.

Images