Projection device and video projection method

Abstract

The application provides a projection device and a video projection method, relates to the technical field of projection, and can solve the problem that the projection device at the present stage cannot meet the demand of users to perform fitness follow-up training while watching fitness videos, and comprises the following steps: in response to a received starting instruction, acquiring regional image information captured by a camera; in the case that the regional image information comprises user image information and the completeness of a user image in the user image information is greater than or equal to a first preset threshold, controlling a projection light machine to project a fitness video; in the case that a user performs fitness follow-up training, determining the face orientation of the user according to fitness action data corresponding to the fitness video; and controlling a driving device to adjust the pose of the projection light machine according to the face orientation of the user. The projection device provided in the application can meet the demand of users to perform fitness follow-up training and improve the use experience of the projection device when the user performs fitness follow-up training.

Description

Technical Field

[0001] This application relates to the field of projection technology, and in particular to a projection device and a video projection method. Background Technology

[0002] Currently, traditional projection devices typically project video onto a fixed projection area, such as a projection screen. If a user changes their activity area while watching the projected video, they need to manually adjust the projection area to continue watching the video. This type of projection device, requiring manual adjustment, has low automation and poor flexibility, failing to provide a good user experience. For example, when a projector is projecting a fitness video and a user is following along, the user's movements will change, and they won't be able to see the video in the projected area in real time. If the user manually adjusts the projection area, it will obviously interrupt their workout, resulting in a poor user experience. Therefore, current projection devices cannot meet the needs of users who want to watch fitness videos while simultaneously following along with their workouts. Summary of the Invention

[0003] To address the aforementioned technical problems, this application provides a projection device and a video projection method to solve the problem that current projection devices cannot meet the needs of users to watch fitness videos while simultaneously engaging in fitness training.

[0004] The technical solution of this application is as follows:

[0005] In a first aspect, this application provides a projection device, comprising: a projection optical engine configured to project a fitness video onto a projection medium; a camera configured to capture regional image information of a viewing area corresponding to the projection device; a driving device configured to adjust the pose of the projection optical engine; and a controller coupled to the projection optical engine, the camera, and the driving device, and configured to: in response to a received start-up command, acquire the regional image information captured by the camera; control the projection optical engine to project the fitness video if the regional image information includes user image information and the completeness of the user image in the user image information is greater than or equal to a first preset threshold; determine the user's face orientation based on fitness movement data corresponding to the fitness video when the user is exercising; and control the driving device to adjust the pose of the projection optical engine based on the user's face orientation.

[0006] The above technical solution has the following advantages or beneficial effects: When projecting fitness videos, this application can determine whether the user is following along and whether they are fully captured by the camera; furthermore, while the user is following along, the projection area of ​​the fitness video is adjusted according to the user's face orientation, allowing the user to view the fitness video completely at any time during the follow-up exercise, thus improving the efficiency and experience of the user's fitness follow-up exercise. Therefore, the technical solution provided by this application allows users to view the projected fitness video completely under different follow-up exercises without manually adjusting the projection area, meeting the user's fitness follow-up exercise needs and improving the user experience of the projection device during fitness follow-up exercise.

[0007] In some embodiments, the driving device is further configured to adjust the pose of the camera; after acquiring the area image information captured by the camera, the controller is further configured to: if the area image information does not include user image information, control the driving device to rotate and adjust the pose of the camera successively in the horizontal direction according to a first angle value until the area image information includes user image information.

[0008] The above technical solution has the following advantages or beneficial effects: When the image information of the area captured by the camera does not include user image information, this application can rotate and adjust the camera's pose to re-capture the viewing area and determine whether a user is present. In this way, even if the user's initial position deviates from the camera's field of view, the user can be brought back into the camera's field of view by adjusting the camera's pose.

[0009] In some embodiments, where the region image information includes user image information, the controller is further configured to: control the driving device to adjust the camera pose sequentially in a direction away from the user according to a first distance threshold when the integrity of the user image in the user image information is less than a first preset threshold, until the integrity of the user image in the user image information is greater than or equal to the first preset threshold.

[0010] The above technical solution has the following advantages or beneficial effects: when the user image information is included in the area image information captured by the camera, but the user image is not complete, this application can adjust the position of the camera in a direction away from the user so that the user can be completely captured by the camera.

[0011] In some embodiments, the controller is further configured to determine a first distance threshold based on the angle between the optical axis of the camera and the horizontal direction, the diagonal field of view, and the height of the focal point above the ground.

[0012] The above technical solution has the following advantages or beneficial effects: by determining the parameters of the camera itself, the size of the adjustment distance is determined each time when adjusting the camera pose in a direction away from the user.

[0013] In some embodiments, after controlling the projector to project the fitness video, the controller is further configured to: determine the degree of matching between user motion data in the user image information and fitness motion data in the fitness video; determine that the user is performing fitness training if the degree of matching is greater than or equal to a second preset threshold; and determine that the user is not performing fitness training if the degree of matching is less than the second preset threshold.

[0014] The above technical solution has the following advantages or beneficial effects: This application can determine whether the user is doing fitness training based on the matching degree between the fitness movement data corresponding to the fitness video and the user movement data captured by the camera, so as to facilitate the smooth progress of subsequent steps.

[0015] In some embodiments, the controller determines the user's face orientation based on the fitness movement data corresponding to the fitness video. Specifically, it is configured to: determine a face orientation data table based on the fitness movement data, the face orientation data table including multiple face orientation data; calculate the proportion of similar data for each face orientation data in the face orientation data table; and determine the face orientation corresponding to the first face orientation data as the user's face orientation if the proportion of similar data for the first face orientation data is greater than or equal to a third preset threshold.

[0016] The above technical solution has the following advantages or beneficial effects: it analyzes and calculates the fitness movement data corresponding to the fitness video, determines the user's face orientation in each set of movements, and provides a reference for the pose adjustment of the projector.

[0017] In some embodiments, the controller controls the drive device to adjust the pose of the projection optical engine based on the user's face orientation. Specifically, the controller is configured to: determine the target orientation of the projection optical engine based on the user's face orientation; adjust the pose of the projection optical engine to project forward when the target orientation is forward, wherein forward is the orientation the user faces towards the viewing area; not adjust the pose of the projection optical engine when the target orientation is backward or downward, wherein backward is the opposite of forward, and downward is the orientation perpendicular to forward and downward in the vertical direction; control the drive device to rotate and adjust the pose of the projection optical engine according to a second angle value when the target orientation is sideways, wherein sideways is the orientation perpendicular to forward and horizontal in the horizontal direction; and control the drive device to rotate and adjust the pose of the projection optical engine according to a third angle value when the target orientation is upward, wherein upward is the opposite of downward.

[0018] The above technical solution has the following advantages or beneficial effects: This application provides a rule for adjusting the pose of the projection optical engine according to the target orientation of the projector when it is turned off, so that the pose adjustment of the projection optical engine can fit the actual application situation.

[0019] In some embodiments, the controller determines the target orientation of the projection engine rotation based on the user's face orientation. Specifically, it is configured to: determine the target orientation to be the same as the user's face orientation when the user's face orientation is forward, sideways, or upward; and determine the target orientation to be forward when the user's face orientation is backward or downward.

[0020] The above technical solution has the following advantages or beneficial effects: This application provides a rule for determining the target orientation of the projector based on the user's face orientation, so that the pose adjustment of the projector can fit the actual application situation.

[0021] In some embodiments, the controller is further configured to determine a second angle value based on the position data of the user image in the user image information, the size of the viewing area in the area image information, and the horizontal field of view of the camera.

[0022] The above technical solution has the following advantages or beneficial effects: by using the parameters contained in the user image information and the parameters of the camera, the angle of rotation for each rotation is determined when adjusting the pose of the projection optical engine.

[0023] In some embodiments, the first angle value is 60 degrees or 90 degrees, and the third angle value is 90 degrees.

[0024] The above technical solution has the following advantages or beneficial effects: the first angle value is set to 60 degrees or 90 degrees, so that the camera can rotate exactly one full circle after rotating an integer number of times; the third angle value is set to 90 degrees, so that after the projector rotates, the fitness video can be projected onto the ceiling above the user, making it convenient for users who raise their heads to watch.

[0025] In some embodiments, the controller is further configured to: acquire the next set of fitness movement data from the fitness video; if the type of movement indicated by the next set of fitness movement data is a lying-down movement, control the drive device to successively adjust the pose of the camera in a direction away from the user according to a first distance threshold until the completeness of the user image in the user image information is greater than or equal to a second preset threshold; and control the drive device to adjust the pose of the projection optical engine according to the user's face orientation.

[0026] The above technical solution has the following advantages or beneficial effects: when the user makes a lying-down exercise, the projection device can automatically adjust the posture so that the user can see the complete projected video and exercise even while lying down.

[0027] In some embodiments, the projection device further includes a speaker, and the controller is coupled to the speaker. The speaker is configured to: play voice; after acquiring the area image information captured by the camera, the controller is further configured to: control the projection optical engine to shut down and control the speaker to play a fitness training interruption voice when the driving device rotates and adjusts the pose of the camera in the horizontal direction according to a first angle value until the projection head rotates one full turn and the area image information captured by the camera does not include user image information.

[0028] The above technical solution has the following advantages or beneficial effects: when the camera rotates one full circle and does not capture the user in the viewing area, it can inform the user that no user exercise has been detected and the exercise session is over.

[0029] In some embodiments, after the driving device successively rotates and adjusts the pose of the camera according to a first angle value until the area image information includes user image information, the controller is further configured to: control the speaker to play a first inquiry voice, the first inquiry voice being used to instruct the user to respond to a first feedback voice command; perform semantic analysis on the first feedback voice command, and control the projection optical engine to continue projecting the fitness video or stop projecting the fitness video based on the result of the semantic analysis.

[0030] The above technical solution has the following advantages or beneficial effects: the projection device can ask the user whether to continue projection after adjusting the camera to capture the complete user image, even if the user's image is not captured initially. The device can also select whether to continue projection based on the user's voice command, thereby improving the user's projection device experience.

[0031] In some embodiments, after determining that the user has not participated in the fitness training, the controller is further configured to: control the speaker to play a second inquiry voice, the second inquiry voice being used to instruct the user to respond to a second feedback voice command; perform semantic analysis on the second feedback voice command, and control the projection optical engine to continue projecting the fitness video or stop projecting the fitness video based on the result of the semantic analysis.

[0032] The above technical solution has the following advantages or beneficial effects: the projection device can ask the user whether to continue projection when it is determined that the user has not been exercising, and select whether to continue projection based on the user's voice command. It can remind the user when they have not been exercising, and eliminates the need for the user to manually turn off the projection when they want to interrupt their exercise, thereby improving the user's experience with the projection device.

[0033] In some embodiments, the projection device further includes a microphone, and a controller is coupled to the microphone. The microphone is configured to: acquire the user's voice; the controller is further configured to: control the microphone to acquire the user's start-up voice command when the start-up command is the user's start-up voice command; control the microphone to acquire the user's first feedback voice command after controlling the speaker to play a first interrogation voice command; and control the microphone to acquire the user's second feedback voice command after controlling the speaker to play a second interrogation voice command.

[0034] The above technical solution has the following advantages or beneficial effects: by setting up a microphone in the projection device, the function of voice collection can be realized.

[0035] Secondly, this application provides a video projection method, comprising: in response to a received start-up command, acquiring area image information captured by a camera; if the area image information includes user image information and the completeness of the user image in the user image information is greater than or equal to a first preset threshold, controlling a projection optical engine to project a fitness video; if the user is exercising, determining the user's face orientation based on the fitness movement data corresponding to the fitness video; and controlling a drive device to adjust the pose of the projection optical engine based on the user's face orientation.

[0036] Thirdly, the present invention provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, the computer program being executed by a processor using a video projection method as provided in any of the second aspects.

[0037] Fourthly, the present invention provides a computer program product that, when run on a computer, causes the computer to perform a video projection method as provided in any of the second aspects.

[0038] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be packaged together with the processor of the display device, or it may be packaged separately from the processor of the display device; this application does not impose any limitation on this.

[0039] The descriptions of the second, third, and fourth aspects in this application can be referenced to the detailed description of the first aspect; and the beneficial effects described in the second, third, and fourth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0040] In this application, the names of the aforementioned display devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those in this application, they fall within the scope of the claims of this application and their equivalents.

[0041] These or other aspects of this application will become more readily apparent in the following description.

[0042] The technical solution provided in this application can determine whether a user is actively following along with the exercise and whether they are fully captured by the camera when projecting fitness videos. Furthermore, while the user is following along, the projection area of ​​the fitness video is adjusted according to the user's facial orientation, allowing the user to view the complete video at any time and improve the efficiency and experience of following along. Moreover, when the user performs a lying-down exercise, the projection device can automatically adjust its posture, ensuring that the user can still see the complete projected video and follow along even while lying down. Therefore, the technical solution provided in this application allows users to view the projected fitness video completely without manually adjusting the projection area, improving the user experience of the projection device during fitness follow-along. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This application provides schematic diagrams of projection scenarios for a projection device according to some embodiments.

[0046] Figure 2 A schematic diagram of the optical path of a projection device provided for some embodiments of this application;

[0047] Figure 3 A schematic diagram of the circuit architecture of a projection device provided for some embodiments of this application;

[0048] Figure 4 A schematic diagram of the optical path of a projection device provided for some embodiments of this application;

[0049] Figure 5 This application provides a schematic diagram of the architecture of a projection device according to some embodiments;

[0050] Figure 6 A schematic diagram of the field of view of a camera provided for some embodiments of this application;

[0051] Figure 7AA flowchart illustrating a video projection method provided for some embodiments of this application;

[0052] Figure 7B A schematic diagram illustrating a process for determining the orientation of a user's face, provided for some embodiments of this application;

[0053] Figure 8 A schematic diagram illustrating the horizontal field of view of a camera provided in some embodiments of this application;

[0054] Figure 9 A schematic diagram of the vertical field of view of a camera provided for some embodiments of this application;

[0055] Figure 10 This is a schematic diagram of the field of view of a camera provided for some embodiments of this application. Detailed Implementation

[0056] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0057] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0058] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0059] The terms “comprising” and “having,” and any variations thereof, are intended to cover but not exclude inclusion. For example, a product or device that includes a series of components is not necessarily limited to all components explicitly listed, but may include other components not explicitly listed or inherent to such product or device. The term “module” means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0060] A projection device is a device that projects media data onto a projection medium. Projection devices can connect to computers, cable TV networks, the internet, VCD (Video Compact Disc), DVD (Digital Versatile Disc Recordable), game consoles, DV camcorders, and other devices via various interfaces to receive the media data to be projected. The media data includes, but is not limited to, images, videos, and text, while the projection medium includes, but is not limited to, physical forms such as walls, screens, and projectors.

[0061] In some embodiments, the projection device may be a laser TV, a projector, or other device with projection capabilities. For example, see [link to laser TV documentation]. Figure 1 A laser TV may include a projection screen 1 and a projector 2. The projection screen 1 is fixed in a first position (such as a TV background wall), and the projector 2 is placed in a second position. By adjusting the relationship between the first and second positions, the projected image of the projector 2 is made to match the projection screen 1, that is, the second position is the optimal placement position of the projector 2.

[0062] In some examples, the projection device may not include a projection screen; for example, projection devices such as laser TVs can project onto projection media such as walls.

[0063] Figure 2 This is a schematic diagram of the optical path of a projection device provided in some embodiments of this application. The light emitted by the laser light source 210 is modulated by the optomechanical system 220 and then imaged on the projection screen 1 through the lens 230.

[0064] In some embodiments, the laser source 210 includes a laser assembly and an optical lens assembly. The beam emitted by the laser assembly can pass through the optical lens assembly to provide illumination for the optomechanical system 220. The optical lens assembly requires a high level of environmental cleanliness and airtightness; while the chamber where the laser assembly is installed can be sealed with a lower level of dustproof sealing to reduce sealing costs.

[0065] In some embodiments, the optical engine 220 may include a blue optical engine, a green optical engine, and a red optical engine, and may also include a heat dissipation system, a circuit control system, etc. The blue, green, and red optical engines together constitute a three-color optical engine.

[0066] In some embodiments, the light-emitting component of a laser TV can also be implemented using a light-emitting diode (LED) light source.

[0067] Figure 3 The circuit architecture diagrams provided for some embodiments of this application are shown.

[0068] In some embodiments, see Figure 3 The projector 2 may include a display control circuit 240, a laser light source 210, at least one laser driver component 250, and at least one brightness sensor 260. The laser light source 210 may include at least one laser corresponding to each of the at least one laser driver component 250. Here, "at least one" refers to one or more, and "more than one" refers to two or more.

[0069] In some embodiments, the laser source 210 includes three lasers corresponding one-to-one with the laser driving assembly 250. These three lasers can be a blue laser 211, a red laser 212, and a green laser 213, respectively. The blue laser 211 emits blue laser light, the red laser 212 emits red laser light, and the green laser 213 emits green laser light. The laser driving assembly 250 can be implemented as including multiple sub-laser driving assemblies, each corresponding to a laser of a different color.

[0070] In some embodiments, the display control circuit 240 is used to output light control signals corresponding to different primary colors to the laser driving assembly 250 to drive the corresponding laser to emit light. For example, the light control signals include blue light control signals, red light control signals, and green light control signals. See also Figure 3 The display control circuit 240 is connected to the laser driver assembly 250 and is used to output at least one light control signal corresponding one-to-one with the three primary colors of each frame in the multi-frame display image, and to transmit the at least one light control signal to the corresponding laser driver assembly 250. For example, the display control circuit 240 can be a microcontroller unit (MCU), also known as a single-chip microcomputer.

[0071] In some embodiments, the laser TV can achieve adaptive adjustment. For example, by setting a brightness sensor 260 in the light output path of the laser light source 210, the brightness sensor 260 can detect a first brightness value of the laser light source 210 and send the first brightness value to the display control circuit 240. The display control circuit 240 can obtain a second brightness value corresponding to the drive current of each laser, and when the difference between the second brightness value and the first brightness value of the laser is greater than a difference threshold, it determines that the laser has a COD (Catastrophic optical damage) fault. Then, the display control circuit 240 can adjust the current control signal of the laser drive component corresponding to the laser until the aforementioned difference is less than or equal to the difference threshold, thereby eliminating the COD fault of the laser, reducing the damage rate of the laser, and improving the image display effect of the projection device.

[0072] Figure 4 This is a schematic diagram of the optical path of a projection device provided in some embodiments of this application.

[0073] In some embodiments, see Figure 4 The optical path structure includes a laser source 210 and optical components 214. The laser source 210 may include independently configured blue laser 211, red laser 212, and green laser 213. This projection device can also be called a three-color projection device. The blue laser 211, red laser 212, and green laser 213 are all modular lightweight (Mirai Console Loader, MCL) packaged lasers, which are small in size and facilitate compact arrangement of the optical path.

[0074] In some embodiments, the projection host 2 may include a controller, which includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory), ROM (Read-Only Memory), a first to an nth interface for input / output, and a communication bus. The controller is connected to relevant hardware of the projection device, such as display control circuitry, a brightness sensor, a distance sensor, and an image acquisition unit, to control the projection device's projection, focusing, calibration, screen on / off state adjustment, and other functions.

[0075] In some embodiments, the projection device (e.g., a laser TV) may be provided with several types of interfaces, such as a power interface, a USB interface, an HDMI (High Definition Multimedia Interface) interface, a network cable interface, a VGA (Video Graphics Array) interface, and a DVI (Digital Visual Interface) interface, to connect to a signal source for transmitting media.

[0076] In some embodiments, after the projection device is started, it can directly enter the display interface of the previously selected signal source, or the signal source selection interface. The signal source can be, for example, a preset video-on-demand program, or one of the following signal sources: HDMI interface, USB interface, live TV interface, etc. After the user selects the target signal source, the projector 2 can acquire media data from the target signal source and project the media data onto the projection screen 1 for display.

[0077] In some embodiments, such as Figure 5The diagram shown is a schematic representation of the architecture of a projection device provided in an embodiment of this application, including a projection optical engine 510, a camera 520, a drive device 530, and a controller 540. This projection device is a self-moving projection robot.

[0078] The projection optical engine 510 is configured to project fitness videos onto a projection medium. In some embodiments, the projection optical engine 510 may be the optical engine 220 described above.

[0079] In some embodiments, the projection medium may be the projection screen 1 described above, or it may be the wall surface of the user's indoor space. For example, in the solution involved in the embodiments of this application, since the pose of the projection optical engine 510 can be changed by the driving device, a fixed projection screen may not be set up, and the fitness video may be directly projected onto the wall surface of the user's indoor space, so that the user can see the projected image when the face is turned.

[0080] In some embodiments, the camera 520 is configured to capture regional image information within the viewing area corresponding to the projection device. Since the user is within the viewing area while performing a fitness workout, the camera 520 captures the regional image information to determine whether the user is within the viewing area, and thus whether to project the fitness video.

[0081] For example, such as Figure 6 As shown, Figure 6 This is a schematic diagram of the field of view of a camera 520 provided in an embodiment of this application. The field of view of the camera 520 is divided into the horizontal field of view (HFOV), the vertical field of view (VFOV), and the diagonal field of view (DFOV). These three field of view angles can be used to calculate the second angle value and the first distance threshold in the following text. For the specific calculation process, please refer to Formula 1, Formula 2, and Formula 3 below.

[0082] exist Figure 6 In the diagram, the horizontal field of view is the angle β formed by line segment 2 and line segment 3; the vertical field of view is the angle α formed by line segment 1 and line segment 2; and the diagonal field of view is the angle γ formed by line segment 1 and line segment 3.

[0083] In some embodiments, the drive device 530 (since it is disposed inside the projection device) Figure 5(Not shown in the image) is configured to adjust the orientation of the projection optical engine 510. It is understood that since the user's face orientation usually changes when performing fitness exercises, the drive device 530 adjusts the orientation of the projection optical engine 510 to change the area where the fitness video is projected, so that the user can still see a relatively complete projection image even if the user's face orientation changes.

[0084] It should be noted that the driving device 530 can adjust the pose of the projection optical engine 510 in several ways. For example, the driving device 530 can directly adjust the pose of the projection optical engine 510 on the projection device, in which case the projection optical engine 510 is mounted on the projection device in a rotatable and / or telescopic manner; or, the driving device 530 can indirectly adjust the pose of the projection optical engine 510 by adjusting the pose of the projection device itself. This application does not limit the way the driving device 530 adjusts the pose of the projection optical engine 510, and in practical applications, multiple methods can be used in combination to enable the driving device 530 to adjust the pose of the projection optical engine 510.

[0085] In some embodiments, the driving device 530 is further configured to adjust the pose of the camera 520. It is understood that when a user arrives at the viewing area wanting to begin fitness training, the user's location may be far or close to the camera 520, resulting in the camera 520 capturing an area image that may not contain a complete user image or may contain an image that is too small. Therefore, the driving device 530 adjusts the pose of the camera 520 to change the distance between the camera 520 and the user, ensuring that the completeness of the user image contained in the area image information is greater than or equal to a first preset threshold. When the completeness of the user image contained in the area image information is greater than or equal to the first preset threshold, the projection device determines that the user has entered the viewing area and that the camera 520 can monitor the user's image information in real time, ensuring the smooth execution of subsequent steps such as determining whether the user is engaging in fitness training and determining the user's facial orientation.

[0086] Similarly, the driving device 530 can adjust the pose of the camera 520 in various ways. For example, the driving device 530 can directly adjust the pose of the camera 520 on the projection device, in which case the camera 520 is mounted on the projection device in a rotatable and / or retractable manner; or, the driving device 530 can indirectly adjust the pose of the camera 520 by adjusting the pose of the projection device itself. This application does not limit the way the driving device 530 adjusts the pose of the camera 520, and in practical applications, multiple methods can be used in combination to enable the driving device 530 to adjust the pose of the camera 520.

[0087] For example, the first preset threshold can be set to 90%, or it can be determined according to the actual situation in the application. This application embodiment does not specifically limit this.

[0088] In some embodiments, the controller 540 (since it is located inside the projection device) Figure 5 (Not shown in the image), coupled to the projector 510, camera 520, and drive unit 530, and configured to control the projector to project fitness videos in response to a received start command; receive area image information captured by the camera; include user image information in the area image information, and the completeness of the user image in the user image information is greater than or equal to a first preset threshold; and, when the user is exercising, determine the user's face orientation based on the user's action data in the user image information; and control the drive unit to adjust the pose of the projector based on the user's face orientation.

[0089] In some embodiments, the projection device may also include a speaker and a microphone. Figure 5 (Not shown in the image), both are coupled to controller 540. The speaker is configured to play voice, enabling the user to receive corresponding voice information from the projection device. The microphone is configured to capture the user's voice, enabling the projection device to acquire corresponding voice commands from the user.

[0090] In some embodiments, the projection device may further include a secondary camera 550 disposed at the lower part of the device. When the projection device needs to move, the secondary camera 550 can provide mapping and pathfinding functions, enabling the projection device to select a route for movement while avoiding collisions with obstacles during the movement.

[0091] In some embodiments, the projection device may further include a display screen 560 disposed on the side of the device facing the user, for displaying the operating parameters of the projection device to the user. For example, the operating parameters may include device usage time, remaining battery power, projection mode information, etc.

[0092] Currently, traditional projection devices typically project video onto a fixed projection screen. If a user changes their activity area while watching the projected video, they need to manually adjust the projection area to continue watching the video in its entirety.

[0093] Projection devices that require manual adjustment of the projection area have low automation and poor flexibility, failing to provide a good user experience. For example, when a projector is projecting a fitness video and a user is following along, the user cannot see the video in the projected area in real time as their movements change. If the user manually adjusts the projection area, it will obviously interrupt their current workout, resulting in a poor user experience. Therefore, current projection devices cannot meet the needs of users who want to watch fitness videos while simultaneously following along with their workouts.

[0094] To address the aforementioned technical problems, this application provides a projection device for projecting and playing fitness videos. The structure and functions of each part of the projection device can be found in the above embodiments. In the following embodiments, a self-moving projection robot is used as an example for illustration, but this does not mean that the projection device involved in this application can only be of this type. Furthermore, based on the projection device shown in the above embodiments, this embodiment further improves some functions of the projection device.

[0095] For example, such as Figure 7A As shown, Figure 7A A video projection method provided in this application embodiment, wherein the controller performs projection playback of a fitness video according to the video projection method, specifically includes the following steps S701-S718:

[0096] S701: In response to the received start broadcast command, acquire the area image information captured by the camera.

[0097] In some embodiments, the start-up command can be a voice command issued by the user, which is collected by a microphone installed in the projection device and transmitted to the controller; the start-up command can also be a remote control. Then, in response to the voice command issued by the user, the controller acquires image information of the area captured by the camera.

[0098] The image information captured by the camera corresponds to the viewing area of ​​the projection device. When a user wants to participate in a fitness workout, the projection device is positioned in front of the user, and the viewing area is the area where the user is located. Therefore, the image information captured by the camera includes the user's image information.

[0099] In some embodiments, after receiving a start command, the controller will launch the fitness video playback interface and pause video playback, and then wait for the user to arrive at the viewing area.

[0100] S702. Based on the regional image information, determine whether there is a user in the viewing area.

[0101] For example, the controller analyzes the area image information to determine whether user image information exists within it. Furthermore, if the area image information includes user image information, the controller determines that a user exists within the viewing area; conversely, if the area image information does not include user image information, the controller determines that no user exists within the viewing area.

[0102] In one possible implementation, the camera captures a video stream over a specific time period while recording image information of the viewing area. For example, the camera can capture video stream data of the viewing area for 15 seconds and define it as the area image information. The specific recording duration of the camera over the viewing area can be determined according to the actual application, and this application embodiment does not impose a specific limitation.

[0103] Furthermore, the controller invokes a preset algorithm to identify whether a user exists in the area image information captured by the camera. The preset algorithm can be a human detection and recognition model algorithm, or other algorithms with recognition capabilities; this embodiment does not specifically limit the specific algorithm used.

[0104] Understandably, since the user's exact location within the viewing area is unpredictable, there's a possibility that the image captured by the camera in the initial pose might not contain the user's image information. Therefore, this step determines whether a user is present within the current viewing area based on the presence of the user's image information. Only if a user is present within the viewing area will the subsequent step of determining whether to project the fitness video based on the completeness of the user's image be executed, preventing the fitness video from playing before the user is in position.

[0105] Accordingly, if there are users in the viewing area, then proceed to step S703; if there are no users in the viewing area, then proceed to step S712.

[0106] S703. Determine whether the completeness of the user image in the user image information is greater than or equal to the first preset threshold.

[0107] In some embodiments, similar to S702 described above, the controller may invoke a preset algorithm to identify the degree of completeness of the user image in the user image information captured by the camera. The preset algorithm may be a human detection and recognition model algorithm, or other algorithms with recognition capabilities; this application embodiment does not specifically limit this.

[0108] For example, the first preset threshold can be set to 90%, or it can be determined according to the actual situation in the application. This application embodiment does not specifically limit this.

[0109] Understandably, since the user's exact position within the viewing area is unpredictable, the user image captured by the camera in the initial pose may not be complete, for example, only showing the user's upper or lower body. Therefore, this step compares the completeness of the user image with a preset threshold to determine whether the user within the viewing area can be fully captured by the camera, or whether most of the user's body is captured. Furthermore, the step of projecting the fitness video will only proceed if the user within the viewing area is fully captured by the camera, or if most of the user's body is captured, thus preventing the fitness video from playing before the user is in position.

[0110] Accordingly, if the completeness of the user image in the user image information is greater than or equal to the first preset threshold, then proceed to step S704; if the completeness of the user image in the user image information is less than the first preset threshold, then proceed to step S713.

[0111] S704, controls the projector to project fitness videos.

[0112] Among them, fitness videos are those that are understood by the public to have fitness teaching or fitness guidance functions, and users can follow along with the corresponding fitness exercises based on watching fitness videos.

[0113] Understandably, when the projection device is a self-moving projection robot, the projection engine projects the fitness video onto the wall of the user's indoor space. This wall can be the wall directly in front of the user or the wall above the user, thus ensuring that the user can see the projected fitness video while exercising.

[0114] It should be noted that, since the projection device in this embodiment can move spontaneously, and the projection optical engine in the projection device can also perform a certain degree of pose adjustment on the projection device, the projection device can adjust the projection area of ​​the fitness video according to the user's face orientation. For the specific process of adjusting the projection area of ​​the fitness video according to the user's face orientation, please refer to S707 below, which will not be repeated here.

[0115] S705. Based on the matching degree between user motion data in user image information and fitness motion data in fitness videos, determine whether the user is doing fitness training.

[0116] In some embodiments, the controller can parse user image information to obtain user motion skeleton point data, then determine user actions based on the user motion skeleton point data, and finally summarize the results into user action data. For example, the controller can employ human skeletal keypoint detection technology to perform pose estimation, motion capture, and other processing on the user image information to obtain user motion skeleton point data. Specific human skeletal keypoint detection technologies can be implemented through methods such as motion capture based on inertial sensors, optical motion capture, and visual human motion capture algorithms; this application embodiment does not impose specific limitations.

[0117] In some embodiments, fitness movement data is a file downloaded synchronously along with the fitness video to the local machine. Specifically, the file corresponding to the fitness movement data is a collection of movement sequences, including the array of movements contained in the fitness video, the movement of skeletal points in each group of movements, the time points corresponding to each movement, configuration files, etc.

[0118] Furthermore, after the aforementioned operations, the controller, upon acquiring user motion data and fitness motion data, calculates the matching degree between the two and compares it with a second preset threshold to determine whether the user is engaging in fitness training.

[0119] Specifically, if the matching degree between the two is greater than or equal to a second preset threshold, it is determined that the user is participating in fitness training; if the matching degree is less than the second preset threshold, it is determined that the user is not participating in fitness training. The second preset threshold can be set to 80%, or it can be determined according to the actual situation in the application; this embodiment does not specifically limit it in this way.

[0120] For example, the method for calculating the matching degree between user action data and fitness action data can employ statistical methods such as mean square error, accuracy, and F1 score; distance-based methods such as Euclidean distance and cosine similarity; and methods specific to certain application scenarios such as score matching, Hamming distance, and edit distance. The specific method depends on the actual application scenario, and this application embodiment does not impose any specific limitations.

[0121] Accordingly, if it is determined that the user is participating in a fitness workout, then proceed to step S706; if it is determined that the user is not participating in a fitness workout, then proceed to step S716.

[0122] S706. Determine the user's face orientation based on the fitness movement data corresponding to the fitness video.

[0123] In some embodiments, the user's facial orientation can be categorized as above, below, in front, behind, left, and right, with left and right collectively referred to as lateral. Specifically, in front refers to the orientation the user faces towards the viewing area, i.e., the direction the user faces when preparing to start a workout and facing the projection device; behind is the opposite orientation; below is the orientation perpendicular to the front and downward in the vertical direction, i.e., the direction the user faces when looking down at the ground; above is the opposite orientation to below, i.e., the direction the user faces when looking up at the ceiling; and lateral is the orientation perpendicular to the front and horizontal in the horizontal direction, i.e., the direction the user faces when shaking their head left or right.

[0124] For example, Figure 7B A flowchart illustrating a method for determining a user's facial orientation based on user action data in user image information provided in this application is shown below. Figure 7B As shown, the controller can determine the user's face orientation through the following steps S706a-S706c.

[0125] S706a. Based on the fitness movement data corresponding to the fitness video, determine the face orientation data table, which includes multiple face orientation data.

[0126] In this step, the controller analyzes the fitness movement data, statistically analyzes the specific facial orientation of the demonstrator in each set of movements, and summarizes this data to obtain a facial orientation data table. This facial orientation data table can be understood as showing the distribution of the demonstrator's facial orientation in six directions when performing a set of fitness movements in a fitness video.

[0127] S706b Calculate the percentage of similar data for each face orientation data item in the face orientation data table.

[0128] Among them, the percentage of similar data is the percentage of face orientation data corresponding to a certain direction in all face orientation data included in this set of actions.

[0129] For example, if for a set of movements A, the distribution of the face orientation of the demonstrator in the fitness video is: 8 times facing forward, 0 times facing backward, 1 time facing left, 1 time facing right, 0 times facing upward, and 0 times facing downward, then the proportion of similar data for the face orientation facing forward is 80%, the proportion of similar data for the face orientation facing left and right is 10%, and the proportion of similar data for the other three directions is 0%.

[0130] S706c, if the proportion of similar data in the first face orientation data is greater than or equal to a third preset threshold, the face orientation corresponding to the first face orientation data is determined as the user's face orientation.

[0131] In some embodiments, the third preset threshold may be set to 80%, or the size of the third preset threshold may be determined according to the actual situation. This application embodiment does not specifically limit this.

[0132] For example, when the third preset threshold is set to 80%, in conjunction with the example of S706b, in a set of actions A, the proportion of similar data of the face orientation data corresponding to the front is 80%, which is equal to the third preset threshold. At this time, the controller determines that in this set of actions A, when the user is doing fitness training, the face orientation of the user should be forward.

[0133] The above describes the process by which the controller determines the user's face orientation based on user action data in the user's image information.

[0134] Understandably, through this S706, the controller can determine the facial orientation that the user should adopt when following the instructions of the fitness video from the corresponding fitness movement data. This allows the controller to adjust the position of the projection engine accordingly in subsequent steps, enabling the user to see the fitness video more completely during fitness training.

[0135] S707: Adjust the position of the projection optical engine by controlling the drive device according to the user's face orientation.

[0136] In some embodiments, the controller first determines the target orientation of the projector based on the user's face orientation, and then adjusts the target orientation of the projector based on the target orientation of the projector.

[0137] For example, the controller determines the target orientation of the projection engine according to the following rules:

[0138] (1) When the user's face is facing forward, to the side, or upward, determine that the target orientation is the same as the user's face orientation;

[0139] (2) If the user’s face is facing backward or downward, determine that the target is facing forward.

[0140] It is understandable that the reason for determining the target orientation to be the same as the user's face orientation in rule (1) is that when the user's face is facing forward, to the side, or upward, the projection area after adjusting the position of the projector is also in front of, above, slightly to the left or right of the user's indoor space. These projection areas are still located on the wall, and the user can watch the fitness video relatively completely. However, in rule (2), when the user's face is facing backward or downward, even if the position of the projector is adjusted so that the projection area is located on the wall behind the user and the ground below, the projection effect of the fitness video will be poor due to the obstruction of the user's body and the ground objects. Therefore, when the user's face is facing backward or downward, the position of the projector is not adjusted to avoid the adverse effects of excessive adjustment of the projector's position on the projection effect.

[0141] In some embodiments, after determining the target orientation of the projection optical engine, the controller may control the drive device to adjust the pose of the projection optical engine according to the following rules:

[0142] (a) With the target facing forward, adjust the pose of the projection optical engine to project forward.

[0143] Understandably, for rule (a), since the initial pose of the projector is to project the fitness video onto the projection area in front of the user, when the target orientation is forward, it is sufficient to keep the projector's pose unchanged. However, when the projector's current pose has been adjusted for other orientations, and the controller determines the target orientation is forward, the controller needs to control the drive device to adjust the projector's pose to project forward.

[0144] (b) When the target is facing backward or downward, do not adjust the pose of the projection engine.

[0145] As described above regarding rule (2), when the user's face is facing backward or downward, even if the position of the projector is adjusted, the user's body and other ground objects will block the view, so the controller will not adjust the position of the projector.

[0146] (c) When the target orientation is to the side, the control drive device rotates and adjusts the pose of the projection optical engine in the direction of the target according to the second angle value.

[0147] Understandably, when the target is oriented to the side, it means that the user's head has made a leftward or rightward movement. At this time, the controller controls the drive device to rotate left or right at a certain angle to adjust the position of the projection optical engine, so that the user can still watch the projection of the fitness video after making the movement of their head to the sides.

[0148] For example, such as Figure 8 As shown, Figure 8 This is a schematic diagram of the horizontal field of view of the camera. (Combined with...) Figure 8 The controller determines the second angle value based on the user image's position data in the user image information, the size of the viewing area in the area image information, and the camera's horizontal field of view. For example, the controller specifically determines the second angle value using the following formula 1:

[0149] DA2=[(W a -W h ) / W a ]*HFOV Formula 1

[0150] Where DA2 represents the second angle value, W a W represents the size of the viewing area. h This represents the location data of the user's image, and HFOV represents the horizontal field of view of the camera.

[0151] (d) When the target is facing upwards, the control drive device rotates and adjusts the pose of the projection optical engine according to the third angle value in the direction of the target.

[0152] Understandably, the target is facing upwards, which means that the user's head has made an upward tilting motion. At this time, the controller controls the drive device to rotate upwards at a certain angle to adjust the position of the projection optical engine, so that the user can still watch the projection of the fitness video after making the tilting motion.

[0153] For example, the third angle value can be set to 90 degrees, so that the projection engine changes the projection area from the wall in front of the user to the ceiling above the user, allowing the user to still see the projection of the fitness video even after looking up. Alternatively, the specific setting of the third angle value can be determined according to the actual application, and this application embodiment does not specifically limit it.

[0154] The above explains the rules by which the controller adjusts the position of the projector's optical engine using the drive device. Therefore, through this step, the controller can adjust the projector's position based on the user's facial orientation, allowing the user to see the fitness video more completely during workout sessions.

[0155] It should be noted that there are multiple ways in which the controller can control the drive device to adjust the pose of the projection optical engine. For example, the drive device can directly adjust the pose of the projection optical engine on the projection device, in which case the projection optical engine is mounted on the projection device in a rotatable and / or retractable manner; or, the drive device can indirectly adjust the pose of the projection optical engine by adjusting the pose of the projection device itself. This application does not limit the method by which the drive device adjusts the pose of the projection optical engine, and in practical applications, multiple methods can be used in combination to enable the drive device to adjust the pose of the projection optical engine. The following description of the adjustment of the projection optical engine pose also refers to this paragraph.

[0156] S708: Obtain the next set of fitness movement data from the fitness video.

[0157] Understandably, the next set of fitness movement data refers to the data corresponding to the next set of fitness movements in the fitness video that follows the current set of movements the user is practicing. For example, since the fitness movement data corresponding to the fitness video is a set of movement sequences, and considering the time points corresponding to each movement in the fitness movement data described in S705 above, the controller can analyze the fitness data file to determine the next set of movements following the user's current fitness movement and obtain the next set of fitness movement data corresponding to that next set.

[0158] S709. Determine whether the movement type indicated by the next set of fitness movement data is a lying-down movement.

[0159] Understandably, when the movement type is a lying-down movement, the corresponding skeletal points of the demonstrator will change significantly. Therefore, the controller can determine whether the movement type indicated by the next set of fitness movement data is a lying-down movement by analyzing the movement of the demonstrator's skeletal points in the next set of fitness movement data.

[0160] Correspondingly, if the controller determines that the next set of fitness movement data indicates a lying-down movement, it executes S710; if the controller determines that the next set of fitness movement data indicates a lying-down movement, it executes S711.

[0161] S710, The control drive device adjusts the camera's pose sequentially in the direction away from the user according to the first distance threshold until the completeness of the user image in the user image information is greater than or equal to the second preset threshold.

[0162] It should be noted that the reason why the controller controls the drive device to adjust the camera's position away from the user is that when the user lies down, the user's entire body will be outside the field of view of the projection device's camera. Therefore, it is necessary to adjust the camera's position so that the camera can capture a more complete image of the user after the position is adjusted.

[0163] In some embodiments, the controller may determine a first distance threshold based on the angle between the camera's optical axis and the horizontal direction, the diagonal field of view, and the focal point height above the ground.

[0164] For example, combined Figure 9 , Figure 9 This is a schematic diagram of the camera's vertical field of view. The specific controller can determine the first distance threshold according to the following formulas 2 and 3:

[0165] DA1 = DFOV / 2 – CA Formula 2

[0166] DF = RH / tan(DA1) (Formula 3)

[0167] Wherein, DA1 represents the angle formed by the lower edge of the vertical field of view and the ground, DFOV represents the diagonal field of view, CA represents the angle between the optical axis of the camera and the horizontal direction, DF represents the first distance threshold, and RH represents the height of the camera's focal point from the ground.

[0168] Furthermore, after determining the first distance threshold, the controller gradually adjusts the camera's pose, moving the camera further away from the user until the camera can capture a relatively complete image of the user. Specifically, the controller determines whether the camera pose adjustment is complete by comparing the completeness of the user image with a second preset threshold.

[0169] For example, the second preset threshold can be set to 90%, or it can be determined according to the actual situation in the application. This application embodiment does not specifically limit this.

[0170] It should be noted that there are multiple ways for the controller to control the drive device to adjust the camera's pose. For example, the drive device can directly adjust the camera's pose on the projection device, in which case the camera is mounted on the projection device in a rotatable and / or retractable manner; or, the drive device can indirectly adjust the camera's pose by adjusting the pose of the projection device itself. This application does not limit the method by which the drive device adjusts the camera's pose, and in practical applications, multiple methods can be used in combination to enable the drive device to adjust the camera's pose. The following description of camera pose adjustment also refers to this paragraph.

[0171] In some embodiments, before the controller adjusts the camera pose sequentially in a direction away from the user according to a first distance threshold, it also records the current original pose of the camera so that the camera pose can be restored to the original pose after the user has completed the next set of fitness exercises.

[0172] S711, does not change the position of the camera.

[0173] Understandably, at this point, the controller determines that the type of movement indicated by the next set of fitness movement data is not a lying-down movement. Therefore, after the user follows the next set of fitness movements, the controller returns to execute S705-S707 to control the drive device to adjust the position of the projection optical engine according to the user's face orientation, so as to ensure that the user can also see the fitness video relatively completely when following the next set of fitness movements.

[0174] S712. The control drive device rotates and adjusts the camera's pose sequentially in the horizontal direction according to the first angle value until the area image information includes the user's image information.

[0175] Understandably, before this step is executed, the area image information does not include user image information. Possible scenarios include: scenario one, the user has not entered the viewing area; scenario two, the user has entered the viewing area but is outside the field of view of the projection device's camera. Therefore, the area image information captured by the camera does not contain user image information.

[0176] For example, with a horizontal field of view of 110 degrees and a vertical field of view of 75 degrees, the horizontal visible range of the camera is 110 degrees. Therefore, the edge of the visible range after rotating the camera 110 degrees exactly overlaps with the edge of the visible range of the camera before rotation. Optionally, for convenience in practical applications, the first angle value can be set to 90 degrees, so that the camera rotates exactly one full circle after 4 rotations; or, the first angle value can be set to 60 degrees, so that the camera rotates exactly one full circle after 6 rotations. The setting of the first angle value depends on the actual application, and this application embodiment does not specifically limit it.

[0177] Optionally, after each rotation of the camera, it re-captures the image information of the viewing area and identifies whether a user exists in the captured image information based on a preset algorithm. The preset algorithm can be a human detection and recognition model algorithm, or other algorithms with recognition capabilities; this embodiment does not specifically limit its application.

[0178] In some embodiments, the controller controls the drive device to rotate horizontally sequentially according to a first angle value to adjust the camera's pose until the projection head has rotated one full circle. If the area image captured by the camera still does not include user image information, the controller then controls the projection optical engine to shut down. Simultaneously, the controller controls the speaker to play a fitness training interruption message. This message indicates to the user that the fitness training session has been interrupted. For example, the message might state, "No user exercise detected; this fitness training session is over."

[0179] Correspondingly, after the controller controls the drive device to rotate and adjust the camera's pose in the horizontal direction according to the first angle value, if the area image information includes user image information, then the controller executes the subsequent S714.

[0180] S713. The control drive device adjusts the camera's pose sequentially in the direction away from the user according to the first distance threshold until the completeness of the user image in the user image information is greater than or equal to the first preset threshold.

[0181] Understandably, before this step, the user image in the area image information may not be complete, for example, only the upper or lower body of the user can be captured. Therefore, the controller can specifically adjust the camera's pose sequentially in a direction away from the user by controlling the drive device until the completeness of the user image in the user image information is greater than or equal to a first preset threshold.

[0182] For example, such as Figure 10 As shown, Figure 10 This relates the lens diameter in millimeters to the corresponding field of view. Figure 10 The diagram shows the field of view of the camera when the lens thickness is 2.8 mm, 3.6 mm, 6 mm, 8 mm, 12 mm, and 16 mm. It can be understood that when the user image is incomplete, moving the camera away from the user to a certain distance will allow the camera to capture a complete image of the user.

[0183] For example, the method for determining the first distance threshold can be found in S710 above. Figure 9 The relevant explanations will not be repeated here.

[0184] It should be noted that after the controller controls the driving device to adjust the camera pose successively in the direction away from the user according to the first distance threshold, if the integrity of the user image in the user image information is greater than or equal to the first preset threshold, then S704 is executed.

[0185] S714, Control the speaker to play the first inquiry voice.

[0186] The first inquiry voice is used to instruct the user to respond with the first feedback voice command.

[0187] For example, the specific content of the first inquiry voice may be "User detected, the current projection position has shifted from the initial position, should we continue playing the projection?", or the specific content of the first inquiry voice may be adjusted according to the actual application. Correspondingly, the user's first feedback voice command can instruct the projection device whether to continue playing the projection. The controller's processing of the first feedback voice command is described in S715 below.

[0188] S715. Perform semantic analysis on the first feedback voice command, and control the projection optical engine to continue projecting the fitness video or stop projecting the fitness video based on the result of the semantic analysis.

[0189] In some embodiments, the controller may perform semantic analysis on the user's first response voice command based on a semantic parsing model.

[0190] For example, the semantic parsing model can be constructed using statistical and machine learning methods, such as the bag-of-words model, the term frequency-inverse document frequency (TF-IDF) model, and topic models; alternatively, the semantic parsing model can also be constructed using neural network methods, including word-to-vector (Word2Vec), bidirectional encoder representations from transformers (BERT), generative pre-trained transformers (GPT), and graph neural networks. In practical applications, appropriate models and techniques can be selected and combined and optimized according to specific needs to construct an efficient semantic parsing model to achieve this step; this application does not specifically limit this approach.

[0191] Furthermore, if the semantic parsing result of the controller's first feedback voice command is "the user instructs to continue projection", the controller controls the projector to continue projecting the fitness video; if the semantic parsing result of the controller's first feedback voice command is "the user instructs to stop projection", the controller controls the projector to stop projecting the fitness video.

[0192] Understandably, based on this step, the projection device can ask the user whether to continue projection even if the user was not initially captured in full, but after adjusting the camera to capture the user in full, and can choose whether to continue projection based on the user's voice command, thereby improving the user's projection device experience.

[0193] S716, Control the speaker to play the second inquiry voice.

[0194] The second inquiry voice is used to instruct the user to respond with a second feedback voice command.

[0195] For example, the specific content of the second inquiry voice could be "Detecting that no fitness training has been performed, should we continue playing the projection?", or the specific content of the second inquiry voice could be adjusted according to the actual application. Correspondingly, the user's second feedback voice command can instruct the projection device whether to continue playing the projection. The controller's processing of the second feedback voice command is described in S717 below.

[0196] S717. Perform semantic analysis on the second feedback voice command, and control the projection optical engine to continue projecting the fitness video or stop projecting the fitness video based on the result of the semantic analysis.

[0197] In some embodiments, the controller may perform semantic analysis on the second feedback voice command responded by the user based on a semantic parsing model.

[0198] For example, the semantic parsing model can be constructed based on statistical and machine learning methods, such as the bag-of-words model, TF-IDF model, and topic model; or, the semantic parsing model can be constructed based on neural network methods, including Word2Vec, BERT, GPT, and graph neural networks. In practical applications, appropriate models and technologies can be selected and combined and optimized according to specific needs to construct an efficient semantic parsing model to achieve this step, and this application embodiment does not specifically limit this.

[0199] Furthermore, if the semantic parsing result of the controller's second feedback voice command is "the user instructs to continue projection", the controller controls the projector to continue projecting the fitness video; if the semantic parsing result of the controller's second feedback voice command is "the user instructs to stop projection", the controller controls the projector to stop projecting the fitness video.

[0200] Understandably, based on this step, the projection device can ask the user whether to continue projection when it determines that the user has not been exercising, and choose whether to continue projection based on the user's voice command. This not only reminds the user when they are not exercising, but also eliminates the need for the user to manually turn off the projection when they want to stop exercising, thereby improving the user's experience with the projection device.

[0201] S718, The fitness video projection playback is complete. Control the projection device to return to its initial position.

[0202] Understandably, after the fitness video projection is finished playing, the controller uses the drive device to return the projection device to its initial position and then turns it off.

[0203] Based on the above technical solution, this application embodiment can determine whether the user is following along with the exercise and whether they are fully captured by the camera when projecting a fitness video. Furthermore, while the user is following along, the projection area of ​​the fitness video is adjusted according to the user's facial orientation, allowing the user to view the complete fitness video at any time, thus improving the efficiency and experience of the exercise. Moreover, when the user performs a lying-down exercise movement, the projection device can automatically adjust its posture, ensuring that the user can still see the complete projected video while lying down. Therefore, the technical solution provided by this application allows users to view the projected fitness video completely without manually adjusting the projection area, improving the user experience of the projection device during exercise.

[0204] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0205] This application also provides a chip system that can be applied to the controller 540 in the foregoing embodiments. The chip system includes at least one processor 2501 and at least one interface circuit 2502. The processor 2501 may be the processor in the controller 540. The processor 2501 and the interface circuit 2502 are interconnected via a line. The processor 2501 can receive and execute computer instructions from the memory of the controller 540 through the interface circuit 2502. When the computer instructions are executed by the processor 2501, the controller 540 can perform the various steps executed by the controller 540 in the foregoing embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in this application.

[0206] Some embodiments of this application also provide a computer-readable storage medium that can store a program. When the computer storage medium is configured in a display device or server, the program, when executed, can include the program steps involved in the image generation method in the above embodiments. The computer storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit it; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0207] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better use the embodiments and various different variations suitable for specific application considerations.

Claims

1. A projection device, characterized in that, include: The projector is configured to project fitness videos onto a projection medium. The camera is configured to capture regional image information of the viewing area corresponding to the projection device; A drive unit is configured to adjust the pose of the projection optical engine; The controller, coupled to the projection optical engine, the camera, and the drive device, is configured to: In response to the received start broadcast command, the image information of the area captured by the camera is acquired; If the area image information includes user image information, and the completeness of the user image in the user image information is greater than or equal to a first preset threshold, the projector is controlled to project the fitness video. When the user is doing fitness training, the user's face orientation is determined based on the fitness movement data corresponding to the fitness video. The driver adjusts the orientation of the projection engine based on the user's facial orientation.

2. The projection device according to claim 1, characterized in that, The driving device is further configured to adjust the pose of the camera; after acquiring the area image information captured by the camera, the controller is further configured to: If the user image information is not included in the area image information, the driving device is controlled to rotate and adjust the position of the camera successively in the horizontal direction according to a first angle value until the user image information is included in the area image information.

3. The projection device according to claim 2, characterized in that, When the user image information is included in the area image information, the controller is further configured to: If the completeness of the user image in the user image information is less than a first preset threshold, the driving device is controlled to adjust the pose of the camera sequentially in a direction away from the user according to a first distance threshold, until the completeness of the user image in the user image information is greater than or equal to the first preset threshold.

4. The projection device according to claim 3, characterized in that, The controller is also configured to: The first distance threshold is determined based on the angle between the optical axis of the camera and the horizontal direction, the diagonal field of view, and the height of the focal point from the ground.

5. The projection device according to claim 4, characterized in that, After controlling the projector to project the fitness video, the controller is further configured to: Determine the degree of matching between the user motion data in the user image information and the fitness motion data in the fitness video; If the matching degree is greater than or equal to the second preset threshold, it is determined that the user is performing the fitness training. If the matching degree is less than a second preset threshold, it is determined that the user has not performed the fitness training.

6. The projection device according to claim 5, characterized in that, The controller determines the user's facial orientation based on the fitness movement data corresponding to the fitness video, specifically configured as follows: Based on the fitness movement data, a face orientation data table is determined, which includes multiple face orientation data. Calculate the percentage of similar data for each face orientation data item in the face orientation data table; If the proportion of similar data in the first face orientation data is greater than or equal to the third preset threshold, the face orientation corresponding to the first face orientation data is determined as the user's face orientation.

7. The projection device according to claim 6, wherein the controller controls the driving device to adjust the pose of the projection optical engine according to the user's facial orientation, specifically configured as follows: The target orientation of the projection engine is determined based on the user's face orientation. When the target is facing forward, the pose of the projection optical engine is adjusted to project towards that direction; wherein, The "front" refers to the direction the user faces when viewing the area. When the target orientation is rearward or downward, the pose of the projection optical engine is not adjusted; wherein, rearward is the opposite orientation to frontward, and downward is the orientation perpendicular to frontward and downward in the vertical direction; When the target orientation is to the side, the drive device is controlled to rotate and adjust the pose of the projection optical engine according to a second angle value in the direction of the target; wherein, the side orientation is perpendicular to the front and in the horizontal direction; When the target orientation is upward, the drive device is controlled to rotate and adjust the pose of the projection optical engine according to a third angle value in the direction of the target orientation; wherein, upward is the opposite orientation to downward.

8. The projection device according to claim 7, characterized in that, The controller determines the target orientation of the projection engine rotation based on the user's facial orientation, specifically configured as follows: If the user's face is facing forward, to the side, or upward, the target orientation is determined to be the same as the user's face orientation. If the user's face is facing backward or downward, the target is determined to be facing forward.

9. The projection device according to claim 8, characterized in that, The controller is also configured to: The second angle value is determined based on the location data of the user image in the user image information, the size of the viewing area in the area image information, and the horizontal field of view of the camera.

10. The projection device according to claim 9, characterized in that, The first angle value is 60 degrees or 90 degrees, and the third angle value is 90 degrees.

11. The projection device according to claim 10, characterized in that, The controller is also configured to: Obtain the next set of fitness movement data from the fitness video; When the next set of fitness movement data indicates a lying-down movement, the drive device is controlled to adjust the position of the camera in a direction away from the user according to the first distance threshold until the completeness of the user image in the user image information is greater than or equal to the second preset threshold. The driver adjusts the orientation of the projection engine based on the user's facial orientation.

12. The projection device according to claim 11, characterized in that, The projection device also includes a speaker, the controller is coupled to the speaker, and the speaker is configured to play voice. After acquiring the area image information captured by the camera, the controller is further configured to: When the driving device rotates and adjusts the position of the camera in the horizontal direction according to the first angle value until the projection head rotates one full turn and the image information of the area captured by the camera does not include the user image information, the projection optical engine is controlled to turn off, and the speaker is controlled to play the fitness training interruption voice.

13. The projection device according to claim 12, characterized in that, After the driving device successively rotates and adjusts the position and pose of the camera according to a first angle value until the area image information includes the user image information, the controller is further configured to: The speaker is controlled to play a first inquiry voice, which is used to instruct the user to respond to a first feedback voice command. The first feedback voice command is semantically analyzed, and the projection optical engine is controlled to continue projecting the fitness video or stop projecting the fitness video based on the result of the semantic analysis.

14. The projection device according to claim 13, characterized in that, After determining that the user has not participated in the fitness training, the controller is further configured to: Control the speaker to play a second inquiry voice, which is used to instruct the user to respond with a second feedback voice command; The second feedback voice command is semantically analyzed, and the projection optical engine is controlled to continue projecting the fitness video or stop projecting the fitness video based on the result of the semantic analysis.

15. The projection device according to claim 14, characterized in that, The projection device also includes a microphone, the controller is coupled to the microphone, and the microphone is configured to collect the user's voice. The controller is also configured to: When the start-up command is the user's start-up voice command, control the microphone to collect the user's start-up voice command; After controlling the speaker to play the first inquiry voice, control the microphone to collect the user's first feedback voice command; After controlling the speaker to play the second interrogation voice, the microphone is controlled to collect the user's second feedback voice command.

16. A video projection method, characterized in that, include: In response to the received start broadcast command, acquire the area image information captured by the camera; If the area image information includes user image information, and the completeness of the user image in the user image information is greater than or equal to a first preset threshold, the projector is controlled to project a fitness video. When a user is following along with a fitness video, the user's facial orientation is determined based on the fitness movement data corresponding to the fitness video. The control drive adjusts the orientation of the projection engine based on the user's facial orientation.

Images