Height adjustment method and device of lifting mechanism and lifting control device of lifting mechanism

Abstract

The application discloses a height adjusting method and device of a lifting mechanism and a lifting control device of the lifting mechanism. The method comprises the following steps: acquiring video stream data of a front area of the lifting mechanism shot by a camera; inputting the video stream data into an object recognition model, and outputting a plurality of cropped images comprising a target object from the object recognition model; inputting the plurality of cropped images into a posture action estimation model, and outputting posture estimation results and action state estimation results of the target object in the plurality of cropped images from the posture action estimation model; determining an action intention of the target object according to the posture estimation results and the action state estimation results of the target object in the plurality of cropped images; and controlling the lifting mechanism to rise or fall according to the action intention of the target object. The application solves the technical problem of wasting manpower caused by manually controlling the lifting mechanism to realize lifting.

Description

Technical Field

[0001] This invention relates to the field of intelligent control, and more specifically, to a height adjustment method, device, and lifting control device for a lifting mechanism. Background Technology

[0002] The monitor adjusts its height according to the user's posture during use, solving a range of problems such as prolonged sitting, standing, and poor posture. It is especially beneficial for people who spend long hours at work, allowing them to easily adjust the monitor height to suit their changing needs, thus improving both their health and work efficiency. It can be widely used in schools, companies, government agencies, and other similar settings.

[0003] Most monitor lifting devices on the market today use buttons to control a motor that lifts the monitor. This approach offers many conveniences by adjusting the monitor's height to match its posture, but it also presents several problems. The time wasted due to frequent manual operation of the monitor's height and the difficulty in controlling the lifting precision under manual operation limit the continued development of this method.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a height adjustment method, device, and lifting control device for a lifting mechanism, to at least solve the technical problem of wasted manpower caused by manually operating the lifting mechanism to achieve lifting.

[0006] According to one aspect of the present invention, a height adjustment method for a lifting mechanism is provided, comprising: acquiring video stream data of a region in front of the lifting mechanism, wherein the video stream data is captured by a camera aimed at the region in front of the lifting mechanism; inputting the video stream data into an object recognition model, wherein the object recognition model outputs a multi-frame cropped image including a target object, wherein the multi-frame cropped image is obtained by cropping an image corresponding to the target object from the video stream data by the object recognition model; inputting the multi-frame cropped image into a posture and motion estimation model, wherein the posture and motion estimation model outputs a posture estimation result and a motion state estimation result of the target object in the multi-frame cropped image; determining the motion intention of the target object based on the posture estimation result and the motion state estimation result of the target object in the multi-frame cropped image; and controlling the lifting mechanism to rise or fall according to the motion intention of the target object.

[0007] Optionally, the step of inputting the video stream data into an object recognition model and outputting a multi-frame cropped image including the target object by the object recognition model includes: inputting the video stream data into a face detection module in the object recognition model, and having the face detection module output facial images of multiple unrecognized objects included in the video stream data; inputting the facial images of the multiple unrecognized objects and the video stream data into the face recognition module in the object recognition model, and outputting the multi-frame cropped image of the target object, wherein the face recognition module determines whether the target object is included among the multiple unrecognized objects based on the facial images of the unrecognized objects, and if the target object is included among the multiple unrecognized objects, crops the human body region image corresponding to the target object to obtain the multi-frame cropped image.

[0008] Optionally, the object recognition model is a model with the following structure: the face detection module is a model based on the RetinaFace face network structure, and the face detection module is trained using a dataset including actions of standing up, sitting down, and leaving; the face recognition module is a model based on the Mobilenetv2 network structure, and the face recognition module is trained using a dataset of feature mimicking.

[0009] Optionally, the step of inputting the multi-frame cropped images into a pose and motion estimation model, and having the pose and motion estimation model output the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images, includes: the pose and motion estimation model determining multiple sets of human key points corresponding one-to-one with the multi-frame cropped images, wherein the multi-frame cropped images are temporally continuous; and the pose and motion estimation model determining the pose estimation result and motion state estimation result of the target object based on the multiple sets of human key points.

[0010] Optionally, the pose estimation model employs a lightweight network, liteHRNet-30, which includes a backbone network, a pose classifier, and a state classifier. The backbone network is used to extract image features from the multi-frame cropped images; the pose classifier is used to determine the pose of the target object corresponding to each of the multi-frame cropped images based on the image features; and the state classifier is used to determine the action state of the target object corresponding to each of the multi-frame cropped images based on the image features.

[0011] Optionally, the state classifier determines the action state of the target object corresponding to each of the multi-frame cropped images through the following steps: determining k frames of images with a sampling interval of m within a time window T in the multi-frame cropped images; the state classifier uses a 2+1D spatiotemporal convolutional neural network to extract the spatial features and temporal features of the k frames of images respectively; determining the context information in the k frames of images based on their respective spatial and temporal features; and determining the action state of the target object corresponding to each of the k frames of images based on the context information and the respective spatial and temporal features of the k frames of images.

[0012] According to another aspect of the present invention, a height adjustment device for a lifting mechanism is also provided, comprising: an acquisition module for acquiring video stream data of a region in front of the lifting mechanism, wherein the video stream data is captured by a camera aimed at the region in front of the lifting mechanism; an identification module for inputting the video stream data into an object recognition model, wherein the object recognition model outputs a multi-frame cropped image including a target object, wherein the multi-frame cropped image is obtained by cropping an image corresponding to the target object from the video stream data by the object recognition model; an estimation module for inputting the multi-frame cropped image into a posture and motion estimation model, wherein the posture and motion estimation model outputs a posture estimation result and a motion state estimation result of the target object in the multi-frame cropped image; a determination module for determining the motion intention of the target object based on the posture estimation result and the motion state estimation result of the target object in the multi-frame cropped image; and a control module for controlling the lifting mechanism to rise or fall according to the motion intention of the target object.

[0013] According to another aspect of the present invention, a lifting mechanism lifting control device is also provided, comprising: a lifting mechanism support assembly, a camera, a lifting drive device, and the aforementioned height adjustment device, wherein the lifting mechanism support assembly is used to support the lifting mechanism and is height-adjustable; the camera is used to capture video stream data of the area in front of the lifting mechanism and transmit the video stream data to the height adjustment device; the lifting drive device is used to drive the lifting mechanism support assembly to rise or fall according to the drive command issued by the height adjustment device.

[0014] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is executed, the device where the non-volatile storage medium is located is controlled to perform the height adjustment method of the lifting mechanism described in any one of the above embodiments.

[0015] According to another aspect of the present invention, a computer device is also provided, the computer device including a memory and a processor, the memory being used to store a program, and the processor being used to run the program stored in the memory, wherein the program, when running, executes the height adjustment method of the lifting mechanism described in any one of the above embodiments.

[0016] In this embodiment of the invention, an object recognition model and a posture and motion estimation model are used to identify the action intention of the target object in front of the lifting mechanism, and the lifting mechanism is controlled to rise and fall according to the action intention of the target object. This achieves the purpose of making the height of the lifting mechanism match the action of the target object in front of the lifting table, thereby realizing the technical effect of automatically adjusting the height of the lifting mechanism without the need for the target object to control it. This solves the technical problem of wasting manpower by manually operating the lifting mechanism to achieve lifting. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a computer terminal for implementing a height adjustment method for a lifting mechanism is shown.

[0019] Figure 2 This is a flowchart illustrating the height adjustment method of the lifting mechanism provided according to an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the structure of a MobileNetV2 network provided according to an optional embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of the structure of the liteHRNet-30 network provided in an optional embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of a 2+1D spatiotemporal convolutional network provided according to an optional embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram of the lifting control logic provided by an optional embodiment of the present invention;

[0024] Figure 7 This is a flowchart of the lifting and adjusting process of the lifting mechanism according to an optional embodiment of the present invention;

[0025] Figure 8 This is a schematic diagram of the lifting and adjusting process of the lifting mechanism according to an optional embodiment of the present invention;

[0026] Figure 9 This is a structural block diagram of the height adjustment device of the lifting mechanism provided in an embodiment of the present invention;

[0027] Figure 10 This is a structural block diagram of a lifting control device for a lifting mechanism according to an embodiment of the present invention;

[0028] Figure 11 This is a schematic diagram of the camera position on a height-adjustable table according to an optional embodiment of the present invention;

[0029] Figure 12 This is a schematic diagram of a lifting drive device for a height-adjustable table according to an optional embodiment of the present invention;

[0030] Figure 13 This is a partial structural schematic diagram of the lifting control device of the lifting mechanism provided in an optional embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] According to an embodiment of the present invention, a method embodiment for adjusting the height of a lifting mechanism is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0034] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a height adjustment method for a lifting mechanism is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the height adjustment method of the lifting mechanism in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the height adjustment method of the lifting mechanism described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0038] Figure 2 This is a flowchart illustrating the height adjustment method of the lifting mechanism provided according to an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes the following steps:

[0039] Step S202: Acquire video stream data of the area in front of the lifting mechanism. The video stream data is captured by a camera positioned in front of the lifting mechanism. The camera needs to capture the user's state and actions using the lifting mechanism; therefore, the camera must be properly positioned so that its shooting area is aligned with the area in front of the lifting mechanism, i.e., the area where the user might be standing. The lifting mechanism can be used to adjust the height of the monitor. For example, the monitor can be supported by the lifting mechanism, and by controlling the raising and lowering of the lifting mechanism, the height of the monitor relative to the user can be adjusted, allowing the user to change posture, such as from standing to sitting, while maintaining the monitor's height at the user's head level. Optionally, the camera can be mounted on the lifting mechanism, on the tabletop of the lifting table where the lifting mechanism and monitor are located, on the support column of the lifting table, on the monitor bracket on the lifting table, or on the base of the lifting mechanism's support bracket.

[0040] Optionally, the camera's field of view can also cover the area where the user of the height-adjustable desk is located. For example, in order to adjust the height of the lifting mechanism on the desk that supports the monitor in order to follow the movements of the person watching the monitor on the desk, the camera can be aimed at the area in front of the lifting mechanism, that is, the area in front of the monitor above the lifting mechanism.

[0041] Optionally, the camera can be placed 50mm above the desktop, 400-600mm from the front of the desktop, and 200mm to the left and right of the center of the human body to collect video data of the user in front of the monitor in real time.

[0042] Step S204: Input the video stream data into the object recognition model, and output a multi-frame cropped image including the target object. The multi-frame cropped image is obtained by cropping the image corresponding to the target object from the video stream data by the object recognition model.

[0043] As an optional embodiment, video stream data is input into an object recognition model, and the object recognition model outputs multi-frame cropped images including the target object. This includes: inputting video stream data into a face detection module in the object recognition model, and the face detection module outputting facial images of multiple unrecognized objects included in the video stream data; inputting the facial images of multiple unrecognized objects and video stream data into a face recognition module in the object recognition model, and outputting multi-frame cropped images of the target object, wherein the face recognition module determines whether the target object is included among the multiple unrecognized objects based on the facial images of the unrecognized objects, and if the target object is included among the multiple unrecognized objects, cropping the human body region image corresponding to the target object to obtain multi-frame cropped images.

[0044] As an optional embodiment, the object recognition model can be constructed in the following manner: a model based on the RetinaFace network structure is selected as the face detection module, and the face detection module is trained using a dataset including standing up, sitting down, and leaving actions; and a model based on the Mobilenetv2 network structure is selected as the face recognition module, and the face recognition module is trained using a dataset based on feature mimicking.

[0045] In the optional embodiments described above, the model used by the face detection module can be customized based on the RetinaFace face network structure and trained on the open-source face dataset Widerface, along with a custom-collected dataset containing actions such as standing up, sitting down, and leaving. To meet the real-time inference requirements of the model, the model complexity can be reduced by using a lightweight network and knowledge distillation for training, thereby improving the model's inference time while keeping the model lightweight.

[0046] Furthermore, the face recognition module used for face recognition employs a model based on the MobileNetv2 network, trained using feature mimicking on the Glint-Asia dataset. This module processes data from video stream X = (x... t x t+1 , ..., x t+k During operation, video frames are cropped based on the facial position of the target object (i.e., the main user). This involves cropping the video into individual images according to the frame rate, and then selecting the image x′ containing the human body region. t+k The clipped portion is used for subsequent pose and state estimation. Once the target object (i.e., the main user) is identified in the video stream, the next steps can proceed.

[0047] Figure 3 This is a schematic diagram of the structure of a MobileNetV2 network provided according to an optional embodiment of the present invention, such as... Figure 3 As shown, when video stream data enters the MobileNetV2 network in the form of images, the data first passes through a 2D convolutional layer and is then input into a self-designed bottleneck layer. The bottleneck layer contains a 3*3 depthwise separable layer, an expansion layer, and a projection layer. Next, it enters a 1*1 2D convolutional layer, and after pooling, it finally passes through a 1*1 2D convolutional layer to obtain the output.

[0048] The above optional embodiments can achieve lightweighting of the models used in face detection, recognition and segmentation, making it easier to place the lightweight models on the computing unit on the side of the lifting table. This allows the processor of the lifting table itself to complete the calculation and determine whether to raise or lower according to the user's intention. All calculations are completed locally, greatly improving the versatility of the product.

[0049] Step S206: Input the multi-frame cropped images into the pose and motion estimation model, and output the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images.

[0050] As an optional embodiment, the multi-frame cropped images are input into the pose and motion estimation model, and the pose and motion estimation model outputs the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images. This includes: the pose and motion estimation model determines multiple sets of human key points that correspond one-to-one with the multi-frame cropped images, wherein the multi-frame cropped images are temporally continuous; and the pose and motion estimation model determines the pose estimation result and motion state estimation result of the target object based on the multiple sets of human key points.

[0051] Optionally, the pose estimation module mainly estimates image frame x′. t+k The location of key points on the human body (e.g., 7 points in total, including 5 points on the face and 2 points on the shoulders) p t+k And based on historical video stream information (x′ t , x′ t+1 ,...,x′ t+k To determine the state s of the target object at time t+k. t+k (Motion and stillness). Historical video stream information refers to the video stream data of the target object captured by the camera at previous moments. The pose of the target object can be estimated based on the position of key points on the human body, and the action of the target object can be estimated based on its state.

[0052] As an optional embodiment, the pose estimation model can employ the lightweight network liteHRNet-30, which includes a backbone network, a pose classifier, and a state classifier. The backbone network is used to extract image features from multiple cropped images; the pose classifier is used to determine the pose of the target object corresponding to each of the multiple cropped images based on the image features; and the state classifier is used to determine the action state of the target object corresponding to each of the multiple cropped images based on the image features.

[0053] Figure 4 This is a schematic diagram of the structure of the liteHRNet-30 network provided in an optional embodiment of the present invention, as shown below. Figure 4 As shown, image data can be input into the liteHRNet-30 network in a 256*256 format. First, it enters a 3*3 2D convolutional network and a shuffle block to extract data features. Then, it passes through three conditional weights and a fusion block specific to this network model, and finally outputs the result.

[0054] As an optional embodiment, the state classifier determines the action state of the target object corresponding to each of the multi-frame cropped images through the following steps: determining k frames of images with a sampling interval of m within a time window of T in the multi-frame cropped images; the state classifier uses a 2+1D spatiotemporal convolutional neural network to extract the spatial features and temporal features of the k frames of images respectively; determining the context information in the k frames of images based on their respective spatial and temporal features; and determining the action state of the target object corresponding to each of the k frames of images based on the context information and the respective spatial and temporal features of the k frames of images.

[0055] Once face detection identifies the object in the video stream data as the target object, i.e., the main user of the height-adjustable desk, the pose and motion estimation model is activated for subsequent processing. This model is mainly used to estimate the pose and state information of the main user at the current moment (the state can include both active and static states). The core of the model contains a multi-task network, and the backbone network of the network (hereinafter referred to as...) Instead, the lightweight network liteHRNet-30 is used to extract image features. The lightweight network liteHRNet-30 is also used for pose estimation. pose (pose classifier) ​​and state classification (f state (State classifier).

[0056]

[0057]

[0058] In the above expression, (x′ t-k+1 ,...,x′ t-1, x′ t Let (x') be k frames with a sampling interval of m within a time window of time T (satisfying m×k=T*FPS). Since state reasoning requires accurately identifying whether the user is moving or stationary, relying solely on single-frame information is insufficient for accurate identification. Therefore, (x') can be extracted. t-k+1 ,...,x′ t-1 , x′ t The system uses k frames of information to identify the user's state. The state classifier can be based on a 2+1D spatiotemporal convolutional neural network. This network decomposes the computation into a 2D convolutional network extracting spatial features from the data, and a 1D convolutional network extracting temporal features from the data, fusing contextual information within a T-time window to accurately identify the user's state. To improve inference efficiency, this module maintains a historical feature queue of size k. Each newly sampled video frame x′... t Features extracted using backbone Afterwards, Remove from queue.

[0059] Figure 5 This is a schematic diagram of a 2+1D spatiotemporal convolutional network provided by an optional embodiment of the present invention. The 2+1D spatiotemporal convolutional network can divide a traditional t*d*d 3D network into a 1*d*d 2D network and a t*1*1 1D network. The 2D network is responsible for extracting the spatial features of the data and fusing the temporal features of the data extracted by the 1D network.

[0060] Step S208: Determine the target object's action intention based on the pose estimation results and action state estimation results of the target object in the multi-frame cropped images.

[0061] After the human posture and state are estimated, the user's state (action) intention can be accurately assessed based on the target object's historical posture, historical state, and current display height information. This will then drive the lift to adjust the height of the lifting table or the height of the display on the lifting table to match the user's sitting / standing posture.

[0062] Step S210: Control the lifting mechanism to rise or fall according to the target object's intention.

[0063] In this step, the lifting mechanism can be the lifting mechanism of the adjustable height-adjustable desk on which the monitor is located. If the adjustable height-adjustable desk has a monitor lifting control device, the height of the lifting mechanism can be adjusted through the monitor lifting control device, thereby adjusting the height of the desk itself and changing the monitor's position in space. Alternatively, if the desk on which the monitor is located is not an adjustable height desk, but the monitor's support bracket has a height-adjustable function, the lifting mechanism can also be a device within the support bracket that controls the lifting. The lifting drive device can control the lifting mechanism within the monitor's support bracket, thereby controlling the monitor's position in space.

[0064] Figure 6 This is a schematic diagram of the lifting control logic provided by an optional embodiment of the present invention, such as... Figure 6 As shown, the core logic of the height control of the adjustable desk is to accurately assess the user's state (action) intent based on the currently identified user posture and state information, as well as historical posture and state information, and drive the lift to adjust the monitor height. When the posture information determines that the user is in front of the adjustable desk, this step will determine the user's state (action) intent based on the current state information, historical state information, and the height control logic diagram (i.e., ... Figure 6 The algorithm determines whether the monitor needs to be raised or lowered. If it does, and the user is determined to be stationary, it assesses the match between the current monitor height and the user's posture, and then drives the lift to match the user's posture. This visual adaptive algorithm solves the inconvenience caused by the reliance on manual operation for traditional monitor height adjustment. It also addresses the issue of other algorithms having large model sizes that cannot be deployed on the hardware platform of height-adjustable desks. Furthermore, it solves the problems of low recognition accuracy, poor real-time performance, and poor inference ability.

[0065] Through the above steps, the object recognition model and posture motion estimation model are used to identify the action intention of the target object in front of the lifting mechanism, and the lifting mechanism is controlled to rise and fall according to the action intention of the target object. This achieves the purpose of matching the height of the lifting mechanism with the action of the target object in front of the lifting mechanism, thereby realizing the technical effect of automatically adjusting the height of the lifting mechanism without the need for the target object to control it. This solves the technical problem of wasting manpower by manually operating the lifting mechanism.

[0066] Figure 7 This is a flowchart of the lifting and adjusting process of the lifting mechanism according to an optional embodiment of the present invention. Figure 8 This is a schematic diagram of the lifting and adjusting process of the lifting mechanism according to an optional embodiment of the present invention. (See Figure 7.) Figure 8As shown, the video stream data (input video) captured by the camera can be input as multiple frames into the face detection and recognition model to determine whether the main user of the height-adjustable table exists in the video stream data; the main user is the target object. If the target object exists, the image of the target object is cropped from the multi-frame images including the target object and input into the human posture and pose recognition model to determine the target object's posture and movement. Furthermore, to accurately identify the target object's posture and movement, information from historical frames including the target object can be referenced to establish a historical pose queue and a historical state queue for the target object. Based on the historical information, the posture and state of the target object in the current video frame are determined, and based on the posture and state of the target object in the current video frame, it is determined whether the lifting mechanism of the height-adjustable table should be raised or lowered, and how to control the raising and lowering.

[0067] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the height adjustment method of the lifting mechanism according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0069] According to an embodiment of the present invention, a height adjustment device for a lifting mechanism for implementing the height adjustment method of the above-described lifting mechanism is also provided. Figure 9 This is a structural block diagram of the height adjustment device of the lifting mechanism according to an embodiment of the present invention, such as... Figure 9 As shown, the height adjustment device 90 of the lifting mechanism includes: an acquisition unit 902, an identification unit 904, an estimation unit 906, a decision unit 908, and a control unit 910. The height adjustment device 90 of the lifting mechanism will be described below.

[0070] Acquisition unit 902 is used to acquire video stream data of the area in front of the lifting mechanism, wherein the video stream data is captured by a camera facing the area in front of the lifting mechanism;

[0071] The recognition unit 904 is used to input video stream data into the object recognition model and output a multi-frame cropped image including the target object. The multi-frame cropped image is obtained by cropping the image corresponding to the target object from the video stream data by the object recognition model.

[0072] The estimation unit 906 is used to input multi-frame cropped images into the pose and motion estimation model, and the pose and motion estimation model outputs the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images.

[0073] Decision unit 908 is used to determine the action intention of the target object based on the pose estimation result and action state estimation result of the target object in the multi-frame cropped images;

[0074] The control unit 910 is used to control the lifting mechanism to rise or fall according to the action intention of the target object.

[0075] It should be noted that the acquisition unit 902, identification unit 904, estimation unit 906, decision-making unit 908, and control unit 910 mentioned above correspond to steps S202 to S210 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in the embodiments.

[0076] According to an embodiment of the present invention, a lifting control device for a lifting mechanism is also provided. Figure 10 This is a structural block diagram of a lifting control device for a lifting mechanism according to an embodiment of the present invention, such as... Figure 10 As shown, the lifting control device 1000 of the lifting mechanism includes: a lifting mechanism support assembly 1001, a camera 1002, a lifting drive device 1003, and a height adjustment device 90. The camera 1002 is used to capture video stream data of the area in front of the lifting mechanism and transmit the video stream data to the height adjustment device 90 of the lifting mechanism. The lifting drive device 1003 is used to drive the lifting mechanism support assembly to rise or fall according to the drive command issued by the height adjustment device 90 of the lifting mechanism.

[0077] Optionally, the lifting mechanism support component includes at least one of the following: a lifting table, a lifting mechanism support bracket; wherein, when the lifting mechanism support component includes a lifting table, the camera is mounted on the tabletop of the lifting table; and when the lifting mechanism support component includes a lifting mechanism support bracket, the camera is mounted on the base of the lifting mechanism support bracket.

[0078] Figure 11 This is a schematic diagram of the camera position of a height-adjustable table according to an optional embodiment of the present invention. Figure 12 This is a schematic diagram of a lifting drive device for a height-adjustable table according to an optional embodiment of the present invention. Figure 11 As shown, the camera can be mounted on the surface of the height-adjustable desk to capture images of the area in front of the desk. Figure 12 As shown, optionally, the lifting drive device may include a base and a lifting column, and may also include a lifting drive motor inside the device (not shown in the figure). The lifting column can serve as a monitor stand to fix the monitor on the lifting table, controlling the monitor to rise or fall to follow the movement of the target object, thus reducing the user's effort in adjusting the height of the lifting table. The camera can be positioned on the front of the base to easily capture images of the target object in front of the monitor. Optionally, the camera can be replaced by other similar sensors, such as an infrared sensor.

[0079] Optionally, when the camera is mounted on the tabletop of the height-adjustable table, the device further includes: an upper housing, a lower housing, a camera mounting bracket, a push-button assembly, a locking plate, and a camera switch. The lower housing is fixed to the tabletop of the height-adjustable table, the upper housing is movably connected to the lower housing via a sliding groove, the camera mounting bracket is used to fix the camera to the upper housing, the push-button assembly is used to control the upper housing to switch between pop-up and retracted states when the upper housing is pressed, and the camera switch is used to control the camera to turn on when popped up and turn off when retracted.

[0080] Figure 13 This is a partial structural schematic diagram of the lifting control device of the lifting mechanism provided in an optional embodiment of the present invention, as shown below. Figure 13 As shown, the camera in the figure could also be other types of sensors used to capture human movements and states in front of the lifting mechanism. Based on Figure 13 The structure shown allows the camera to be retracted below the height-adjustable tabletop. When the user presses the push-button assembly through the upper housing, the camera rises above the tabletop and is turned on by the camera switch. At this time, the camera captures the area in front of the monitor and sends the video stream data to subsequent devices for analysis. When the user presses the upper housing again, the camera can be turned off by the camera switch and retracted below the tabletop. This increases the utilization of the desktop and avoids the wear and tear on the camera caused by prolonged operation.

[0081] Embodiments of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0082] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the height adjustment method and device of the lifting mechanism in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the height adjustment method of the lifting mechanism described above. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0083] The processor can access information and applications stored in memory via a transmission device to perform the following steps: acquiring video stream data of the area in front of the lifting mechanism, wherein the video stream data is captured by a camera corresponding to the lifting mechanism; inputting the video stream data into an object recognition model, which outputs a multi-frame cropped image including the target object, wherein the multi-frame cropped image is obtained by cropping the image corresponding to the target object from the video stream data; inputting the multi-frame cropped image into a posture and motion estimation model, which outputs the posture estimation result and motion state estimation result of the target object in the multi-frame cropped image; determining the target object's motion intention based on the posture estimation result and motion state estimation result of the target object in the multi-frame cropped image; and controlling the lifting mechanism to rise or fall according to the target object's motion intention.

[0084] Optionally, the processor may also execute program code for the following steps: inputting video stream data into an object recognition model, and outputting multi-frame cropped images including the target object from the object recognition model, including: inputting video stream data into a face detection module in the object recognition model, and outputting facial images of multiple unrecognized objects included in the video stream data from the face detection module; inputting the facial images of multiple unrecognized objects and video stream data into a face recognition module in the object recognition model, and outputting multi-frame cropped images of the target object, wherein the face recognition module determines whether the target object is included among the multiple unrecognized objects based on the facial images of the unrecognized objects, and if the target object is included among the multiple unrecognized objects, cropping the human body region image corresponding to the target object to obtain multi-frame cropped images.

[0085] Optionally, the processor may also execute program code with the following steps: the object recognition model is a model with the following structure: the face detection module is a model based on the RetinaFace face network structure, and the face detection module is trained using a dataset that includes actions such as standing up, sitting down, and leaving; the face recognition module is a model based on the Mobilenetv2 network structure, and the face recognition module is trained using a dataset that simulates feature mimicking.

[0086] Optionally, the processor may also execute program code that performs the following steps: inputting multi-frame cropped images into a pose and motion estimation model, and having the pose and motion estimation model output the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images, including: the pose and motion estimation model determining multiple sets of human key points that correspond one-to-one with the multi-frame cropped images, wherein the multi-frame cropped images are temporally continuous; and the pose and motion estimation model determining the pose estimation result and motion state estimation result of the target object based on the multiple sets of human key points.

[0087] Optionally, the processor may also execute program code for the following steps: The pose estimation model uses a lightweight network liteHRNet-30, which includes a backbone network, a pose classifier, and a state classifier. The backbone network is used to extract image features from multiple cropped images; the pose classifier is used to determine the pose of the target object corresponding to each of the multiple cropped images based on the image features; and the state classifier is used to determine the action state of the target object corresponding to each of the multiple cropped images based on the image features.

[0088] Optionally, the processor may also execute program code with the following steps: The state classifier determines the action state of the target object corresponding to each of the multi-frame cropped images through the following steps: determining k frames of images with a sampling interval of m within a time window of T in the multi-frame cropped images; the state classifier uses a 2+1D spatiotemporal convolutional neural network to extract the spatial features and temporal features of the k frames of images respectively; determining the context information in the k frames of images based on their respective spatial and temporal features; and determining the action state of the target object corresponding to each of the k frames of images based on the context information and the respective spatial and temporal features of the k frames of images.

[0089] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a non-volatile storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0090] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the aforementioned non-volatile storage medium can be used to store the program code executed by the height adjustment method of the lifting mechanism provided in the above embodiments.

[0091] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0092] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring video stream data of the area in front of the lifting mechanism, wherein the video stream data is captured by a camera corresponding to the lifting mechanism; inputting the video stream data into an object recognition model, and having the object recognition model output multi-frame cropped images including the target object, wherein the multi-frame cropped images are obtained by cropping the image corresponding to the target object from the video stream data by the object recognition model; inputting the multi-frame cropped images into a posture and motion estimation model, and having the posture and motion estimation model output the posture estimation result and motion state estimation result of the target object in the multi-frame cropped images; determining the motion intention of the target object based on the posture estimation result and motion state estimation result of the target object in the multi-frame cropped images; and controlling the lifting mechanism to rise or fall according to the motion intention of the target object.

[0093] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: inputting video stream data into an object recognition model, and outputting multi-frame cropped images including the target object from the object recognition model, including: inputting video stream data into a face detection module in the object recognition model, and outputting facial images of multiple unrecognized objects included in the video stream data from the face detection module; inputting the facial images of multiple unrecognized objects and video stream data into a face recognition module in the object recognition model, and outputting multi-frame cropped images of the target object, wherein the face recognition module determines whether the target object is included among the multiple unrecognized objects based on the facial images of the unrecognized objects, and if the target object is included among the multiple unrecognized objects, cropping the human body region image corresponding to the target object to obtain multi-frame cropped images.

[0094] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: the object recognition model is a model with the following structure: the face detection module is a model based on the RetinaFace face network structure, and the face detection module is trained using a dataset including actions of standing up, sitting down, and leaving; the face recognition module is a model based on the Mobilenetv2 network structure, and the face recognition module is trained using a dataset based on feature mimicking.

[0095] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: inputting multi-frame cropped images into a pose and motion estimation model, and outputting the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images by the pose and motion estimation model, including: the pose and motion estimation model determining multiple sets of human key points corresponding one-to-one with the multi-frame cropped images, wherein the multi-frame cropped images are temporally continuous; the pose and motion estimation model determining the pose estimation result and motion state estimation result of the target object based on the multiple sets of human key points.

[0096] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: the pose estimation model uses a lightweight network liteHRNet-30, which includes a backbone network, a pose classifier, and a state classifier. The backbone network is used to extract image features from multiple cropped images; the pose classifier is used to determine the pose of the target object corresponding to each of the multiple cropped images based on the image features; and the state classifier is used to determine the action state of the target object corresponding to each of the multiple cropped images based on the image features.

[0097] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: The state classifier determines the action state of the target object corresponding to each of the multi-frame cropped images through the following steps: determining k frames of images with a sampling interval of m within a time window T in the multi-frame cropped images; the state classifier uses a 2+1D spatiotemporal convolutional neural network to extract the spatial features and temporal features of the k frames of images respectively; determining the context information in the k frames of images based on their respective spatial and temporal features; and determining the action state of the target object corresponding to each of the k frames of images based on the context information and the respective spatial and temporal features of the k frames of images.

[0098] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0099] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0104] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for adjusting the height of a lifting mechanism, characterized in that, include: Acquire video stream data of the area in front of the lifting mechanism, wherein the video stream data is captured by a camera facing the area in front of the lifting mechanism; The video stream data is input into an object recognition model, which outputs a multi-frame cropped image including the target object. The multi-frame cropped image is obtained by cropping the image corresponding to the target object from the video stream data by the object recognition model. The object recognition model includes a face detection module and a face recognition module. The method further includes: detecting facial images of multiple unrecognized objects from the video stream data by the face detection module; determining whether the multiple unrecognized objects include the pre-registered target object based on the facial images of the multiple unrecognized objects by the face recognition module; and cropping the human body region image corresponding to the target object from the video stream data if the multiple unrecognized objects include the target object, thus forming a multi-frame cropped image with temporal continuity. The method further includes: inputting the multi-frame cropped images into a pose estimation model, which outputs the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images; the pose estimation model employs a lightweight network, which includes a backbone network, a pose classifier, and a state classifier; extracting image features from the multi-frame cropped images using the backbone network; determining the static pose of the target object corresponding to each of the multi-frame cropped images based on the image features using the pose classifier; and using a 2+1D spatiotemporal convolutional neural network, extracting spatial and temporal features from k frames of images with a sampling interval of m within a time window of T, determining contextual information based on the spatial and temporal features of each of the k frames, and then determining the motion state of the target object corresponding to each of the k frames, wherein the motion state includes movement and stillness. Based on the pose estimation results and motion state estimation results of the target object in the multi-frame cropped images, the motion intention of the target object is determined; based on the motion intention of the target object, the lifting mechanism is controlled to rise or fall.

2. The method according to claim 1, characterized in that, The object recognition model has the following structure: The face detection module is a model based on the RetinaFace face network structure. The face detection module is trained using a dataset that includes standing up, sitting down, and leaving actions. The facial recognition module is a model based on the MobileNetv2 network structure, and it is trained on a dataset that simulates feature mimicking.

3. The method according to claim 1, characterized in that, The step of inputting the multi-frame cropped images into the pose and motion estimation model, and having the pose and motion estimation model output the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images, includes: The pose estimation model determines multiple sets of human body key points that correspond one-to-one with the multi-frame cropped images, wherein the multi-frame cropped images are temporally continuous. The posture and motion estimation model determines the posture estimation result and motion state estimation result of the target object based on the multiple sets of human key points.

4. A height adjustment device for a lifting mechanism, characterized in that, include: An acquisition unit is used to acquire video stream data of the area in front of the lifting mechanism, wherein the video stream data is captured by a camera facing the area in front of the lifting mechanism; The recognition unit is used to input the video stream data into an object recognition model, and the object recognition model outputs a multi-frame cropped image including the target object. The multi-frame cropped image is obtained by cropping the image corresponding to the target object from the video stream data by the object recognition model. The object recognition model includes a face detection module and a face recognition module. The face detection module detects facial images of multiple unrecognized objects from the video stream data. The face recognition module determines whether the multiple unrecognized objects include the pre-registered target object based on the facial images of the multiple unrecognized objects. If the target object is determined to be included among the multiple unrecognized objects, the human body region image corresponding to the target object is cropped from the video stream data to form a temporally continuous multi-frame cropped image. An estimation unit is used to input the multi-frame cropped images into a pose and motion estimation model, and the pose and motion estimation model outputs the pose estimation result and motion state estimation result of the target object in the multi-frame cropped images. The pose and motion estimation model uses a lightweight network, which includes a backbone network, a pose classifier, and a state classifier. The backbone network extracts image features from the multi-frame cropped images; the pose classifier determines the static pose of the target object corresponding to each of the multi-frame cropped images based on the image features; the state classifier uses a 2+1D spatiotemporal convolutional neural network to extract spatial and temporal features from k frames of images with a sampling interval of m within a time window of T. Context information is determined based on the spatial and temporal features of each of the k frames, thereby determining the motion state of the target object corresponding to each of the k frames. The motion state includes movement and stillness. The decision unit is used to determine the action intention of the target object based on the pose estimation result and the action state estimation result of the target object in the multi-frame cropped image; The control unit is used to control the lifting mechanism to rise or fall according to the action intention of the target object.

5. A lifting control device for a lifting mechanism, characterized in that, The lifting mechanism lifting control device includes: a lifting mechanism support component, a camera, a lifting drive device, and a height adjustment device as described in claim 4, wherein the lifting mechanism support component is used to support the lifting mechanism and its height is adjustable; the camera is used to capture video stream data of the area in front of the lifting mechanism and transmit the video stream data to the height adjustment device; the lifting drive device is used to drive the lifting mechanism support component to rise or fall according to the drive command issued by the height adjustment device.

6. The apparatus according to claim 4, characterized in that, The face detection module is a model based on the RetinaFace face network structure, and the face detection module is trained using a dataset that includes standing up, sitting down, and leaving actions; the face recognition module is a model based on the MobileNetv2 network structure, and the face recognition module is trained using a dataset that simulates features.

7. The apparatus according to claim 5, characterized in that, The estimation unit includes: The key point subunit is used by the pose and motion estimation model to determine multiple sets of human body key points that correspond one-to-one with the multi-frame cropped images, wherein the multi-frame cropped images are temporally continuous. The sub-unit is determined, and the posture and motion estimation model determines the posture estimation result and motion state estimation result of the target object based on the multiple sets of human key points.

8. The apparatus according to claim 5, characterized in that, The control unit includes: A signal generation subunit is used to generate a drive command based on the action intention of the target object, wherein the drive command is used to control the lifting drive device to perform an action; The signal transmission subunit is used to transmit the drive command to the lifting drive device.

9. The apparatus according to claim 5, characterized in that, The lifting mechanism support assembly includes at least one of the following: a lifting table, and a lifting mechanism support frame; wherein... When the lifting mechanism support assembly includes the lifting table, the camera is mounted on the tabletop of the lifting table; When the lifting mechanism support assembly includes the lifting mechanism support bracket, the camera is mounted on the base of the lifting mechanism support bracket.

10. The apparatus according to claim 9, characterized in that, When the camera is mounted on the tabletop of the lifting table, the device further includes: an upper housing, a lower housing, a camera fixing component, a push-button assembly, a locking plate, and a camera switch. The lower housing is fixed to the tabletop of the lifting table, and the upper housing is movably connected to the lower housing via a sliding groove. The camera fixing component is used to fix the camera to the upper housing. The push-button assembly is used to control the upper housing to switch between pop-up and retracted states when it is pressed. The camera switch is used to control the camera to turn on when popped up and turn off when retracted.

11. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the height adjustment method of the lifting mechanism according to any one of claims 1 to 3.

12. A computer device, characterized in that, The computer device includes a memory and a processor. The memory is used to store a program, and the processor is used to run the program stored in the memory. When the program runs, it executes the height adjustment method of the lifting mechanism according to any one of claims 1 to 3.

Images