Adjustment method and device based on multi-modal fusion strategy, computer equipment, readable storage medium and program product
Through a multimodal fusion strategy, combining image and contact pressure information, extracting and fusion characteristics to generate more accurate adjustment parameters, the problem of low adjustment accuracy in traditional technology is solved and the position adjustment of higher accuracy is achieved.
Patent Information
- Application Number
- CN202411358573.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-01-14
AI Technical Summary
In traditional technology, the accuracy of automatically adjusting the supporting facilities through image determination of user node parameters is not high enough, resulting in insufficient adjustment.
Using a method based on a multimodal fusion strategy, combining image information and contact pressure information, features are extracted through multi-dimensional convolutional layer and one-dimensional convolutional layer, and feature fusion is performed using attention network layer to generate more accurate adjustment parameters.
Improve the accuracy of item position adjustment in the target area, ensure higher adjustment accuracy, and adapt to the posture of different target objects.
Smart Images
Figure CN119312279A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an adjustment method, apparatus, computer equipment, computer-readable storage medium and computer program product based on a multimodal fusion strategy. Background Art
[0002] With the widespread development of artificial intelligence technology, machine learning model technology has emerged and machine learning models have been widely used.
[0003] In traditional technology, when a user enters a bank's self-service area, it is necessary to manually adjust the supporting facilities in the area, such as the height of the table and chair, or use an image acquisition device to capture images of users entering the self-service area, extract user features based on the images, and determine the parameters of the user's joints, such as wrists, elbows, knees, ankles, eyes, head, etc. The supporting facilities in the area are automatically adjusted based on the parameters of the joints.
[0004] However, the current adjustment method only determines the user's joint point parameters through images, and there is a problem that the adjustment accuracy of the supporting facilities is not high enough. Summary of the invention
[0005] Based on this, it is necessary to provide an adjustment method, device, computer equipment, computer-readable storage medium and computer program product based on a multimodal fusion strategy that can improve the accuracy of position adjustment of objects in the target area in response to the above technical problems.
[0006] In a first aspect, the present application provides an adjustment method based on a multimodal fusion strategy, the method comprising:
[0007] Identify the target object within the target area;
[0008] Acquire image information of the target object through an image acquisition device; acquire contact pressure information of the target object within a target area through a pressure sensor;
[0009] The image information and the contact pressure information are input into a trained target posture adjustment model, and the adjustment parameters of each object in the target area output by the target posture adjustment model are obtained, wherein the target posture adjustment model is trained based on sample image information and sample contact pressure information through a multimodal fusion strategy.
[0010] In one embodiment, the step of inputting the image information and the contact pressure information into a trained target posture adjustment model, and obtaining adjustment parameters of each object in the target area output by the target posture adjustment model, includes:
[0011] Extracting multi-level image features of the image information through a multi-dimensional convolution layer to obtain joint point features of the human body posture; extracting features of the contact pressure information through a one-dimensional convolution layer to obtain global pressure intensity features;
[0012] Determine the position vector of the joint point of the human body posture according to the joint point features of the human body posture through the posture estimation layer;
[0013] The position vector and the global pressure intensity feature are fused through an attention network layer to obtain a multimodal feature of the human body posture;
[0014] The output layer outputs adjustment parameters of each object in the target area according to the multimodal features of the human body posture.
[0015] In one embodiment, the position vector and the global pressure intensity feature are fused through an attention network layer to obtain a multimodal feature of a human body posture, including:
[0016] For the position vector, generate a first query vector, a first key vector and a first value vector through linear transformation;
[0017] For the global pressure intensity feature, generate a second query vector, a second key vector and a second value vector through linear transformation;
[0018] Determining a first relevance score by performing a dot product on the first query vector and the second key vector;
[0019] Determine a second relevance score by performing a dot product on the second query vector and the first key vector;
[0020] Determine a first weight vector for the position vector and a second weight vector for the global pressure intensity feature by using the first correlation score and the second correlation score;
[0021] A multimodal feature of a human body posture is determined according to the first value vector, the first weight vector, the second value vector, and the second weight vector.
[0022] In one embodiment, extracting the features of the contact pressure information through a one-dimensional convolutional layer to obtain the global pressure intensity features includes:
[0023] Extracting features of the contact pressure information to obtain pressure intensity features at different positions;
[0024] Based on the pressure intensity characteristics at different positions and the corresponding weights, the global pressure intensity characteristics of each object are determined.
[0025] In one embodiment, before inputting the image information and the contact pressure information into the trained target posture adjustment model, the method further includes:
[0026] Acquiring sample image information of the sample object and sample contact pressure information of the sample object on an object in a target area;
[0027] The initial posture adjustment model is trained using the sample image information and the sample contact pressure information to obtain a target posture adjustment model.
[0028] In one embodiment, after inputting the image information and the contact pressure information into a trained target posture adjustment model and obtaining the adjustment parameters of each object in the target area output by the target posture adjustment model, the method further includes:
[0029] When the target object keeps the same posture within the target time period, the voice information is output; and the adjustment parameters corresponding to each of the items are adjusted, and new adjustment parameters corresponding to each of the items are output.
[0030] In a second aspect, the present application further provides an adjustment device based on a multimodal fusion strategy, the device comprising:
[0031] A recognition module, used for identifying a target object in a target area;
[0032] An acquisition module is used to acquire image information of the target object through an image acquisition device; and acquire contact pressure information of the target object on an object in a target area through a pressure sensor;
[0033] A processing module is used to input the image information and the contact pressure information into a trained target posture adjustment model, and obtain the adjustment parameters of each object in the target area output by the target posture adjustment model, wherein the target posture adjustment model is trained based on sample image information and sample contact pressure information through a multimodal fusion strategy.
[0034] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.
[0036] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.
[0037] The above adjustment method, device, computer equipment, computer readable storage medium and computer program product based on the multimodal fusion strategy first identify the target object entering the target area, collect information of multiple modes, such as the image information of the target object and the contact pressure information of the target object in the target area; then input the image information and contact pressure information of the two modes into the trained target posture adjustment model, wherein the target posture adjustment model is trained based on the sample image information and the sample contact pressure information through the multimodal fusion strategy. Finally, the adjustment parameters of each object in the target area output by the target posture adjustment model are obtained. Since one modality of the image information of the target object is collected, another modality of the contact pressure information of the target object in the target area is also collected; and the target posture adjustment model trained by the multimodal fusion strategy extracts and fuses the image information and the contact pressure information, the adjustment parameters of each object in the target area output by the target posture adjustment model are more accurate compared with the information of a single modality. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0039] Figure 1 is an application environment diagram of an adjustment method based on a multimodal fusion strategy in an embodiment;
[0040] Figure 2 is a flow chart of an adjustment method based on a multimodal fusion strategy in one embodiment;
[0041] Figure 3 A schematic diagram of a process of processing information of a target posture adjustment model in one embodiment;
[0042] Figure 4 A schematic diagram of a flow chart of information processing by an attention network layer in one embodiment;
[0043] Figure 5 A schematic diagram of a process of extracting features using a one-dimensional convolutional layer in one embodiment;
[0044] Figure 6 A schematic diagram of a process for training a target posture adjustment model in one embodiment;
[0045] Figure 7 is a structural block diagram of an adjustment device based on a multimodal fusion strategy in an embodiment;
[0046] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0048] The adjustment method based on the multimodal fusion strategy provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 identifies the target object in the target area; obtains the image information of the target object through the image acquisition device; obtains the contact pressure information of the target object in the target area through the pressure sensor; inputs the image information and the contact pressure information into the trained target posture adjustment model, and obtains the adjustment parameters of each object in the target area output by the target posture adjustment model, wherein the target posture adjustment model is trained based on the sample image information and the sample contact pressure information through the multimodal fusion strategy. Among them, the terminal 102 can be, but is not limited to, smart tables and chairs, smart lamps, smart bookshelves, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0049] In an exemplary embodiment, Figure 2 As shown in FIG, an adjustment method based on a multimodal fusion strategy is provided, and this method is applied to Figure 1 The server in the example is used to illustrate, including the following steps S202 to S206. Among them:
[0050] Step S202: identifying a target object in the target area.
[0051] The target area is a pre-demarcated area, which can be a business processing area, an office, an open area, or a closed area.
[0052] Optionally, when a target object enters a target area, for example, a user who needs to handle business enters a business handling area, the server will identify the person entering the office or the business handling area.
[0053] Step S204, acquiring image information of the target object through an image acquisition device; acquiring contact pressure information of the target object on an object in a target area through a pressure sensor.
[0054] The image acquisition device may be a video camera, a still camera, an infrared device, etc. The pressure sensor may be a gravity sensor.
[0055] Optionally, the server obtains image information of the target object through an image acquisition device; for example, the image acquisition device can capture a video of the target object entering the target area or several initial images; the image acquisition device sends the captured video or several initial images to the server; the server receives the video or several initial images, and performs standardization processing on them to obtain image information.
[0056] The pressure sensor obtains the contact pressure information of the target object in the target area, such as the weight of the target object entering the target area, or the pressure of the target object's hands on the smart desk, or the pressure of the target object's feet on the ground when the target object is in the smart chair, or extracts the contact point information of the target object from the pressure sensor data on the chair and the table to determine the user's center of gravity distribution, whether there is uneven pressure concentration, etc. The pressure sensor sends the collected contact pressure information to the server.
[0057] Step S206: input the image information and the contact pressure information into the trained target posture adjustment model, and obtain the adjustment parameters of each object in the target area output by the target posture adjustment model.
[0058] Among them, the target posture adjustment model is trained based on sample image information and sample contact pressure information through a multimodal fusion strategy.
[0059] Optionally, the server inputs the image information and contact pressure information into a trained target posture adjustment model. After the target posture adjustment model obtains the image information and contact pressure information, the target posture adjustment model extracts high-level posture features through a convolutional neural network, and the data of the pressure sensor is processed through a one-dimensional convolutional network or a fully connected network to extract pressure distribution features. In the middle layer of the network, these two types of features are fused, and the fused features can simultaneously consider the user's posture and contact pressure. The target posture adjustment model passes through the output layer: the fused feature vector is input into a decision network, which outputs the adjustment parameters of each item, including the height of the table and chair, the backrest angle, the distance between the table and chair, etc. The server controls the position adjustment of the item according to the adjustment parameters.
[0060] In the above adjustment method based on multimodal fusion strategy, the target object entering the target area is first identified, and information of multiple modes is collected, such as the image information of the target object and the contact pressure information of the target object in the target area; then the image information and contact pressure information of the two modes are input into the trained target posture adjustment model, wherein the target posture adjustment model is trained based on the sample image information and the sample contact pressure information through the multimodal fusion strategy. Finally, the adjustment parameters of each object in the target area output by the target posture adjustment model are obtained. Since the image information of the target object is collected in one mode, the contact pressure information of the target object in the target area is collected in another mode; and the target posture adjustment model trained by the multimodal fusion strategy extracts and fuses the image information and the contact pressure information, the adjustment parameters of each object in the target area output by the target posture adjustment model are more accurate compared with the information of a single mode.
[0061] In an exemplary embodiment, Figure 3 As shown, the image information and contact pressure information are input into the trained target posture adjustment model, and the adjustment parameters of each object in the target area output by the target posture adjustment model are obtained, including steps S302 to S308.
[0062] Step S302, extracting multi-level image features of image information through a multi-dimensional convolution layer to obtain joint point features of human body posture; extracting features of contact pressure information through a one-dimensional convolution layer to obtain global pressure intensity features.
[0063] Among them, the target posture adjustment model includes several convolutional layers, including multi-dimensional convolutional layers and one-dimensional convolutional layers, posture estimation layers, attention network layers, and output layers.
[0064] Optionally, the image information is subjected to multi-level image feature extraction through a multi-dimensional convolutional layer (2D-CNN, 3D-CNN) to obtain joint features of the human body posture, such as shoulders, elbows, knees, etc.
[0065] At the same time, the features of contact pressure information are extracted through a one-dimensional convolutional layer (1D-CNN) to first obtain the pressure intensity features at different positions, and then determine the global pressure intensity features based on the pressure intensity features at different positions.
[0066] Step S304, determining the position vector of the joint point of the human body posture according to the joint point features of the human body posture through the posture estimation layer.
[0067] Optionally, the multi-dimensional convolutional layer extracts multi-level image features, and the obtained joint point features of the human body posture are sent to the posture estimation layer, such as PartAffinity Fields in OpenPose, to finally obtain the position vector of the 2D joint point.
[0068] Step S306, the position vector and the global pressure intensity feature are fused through the attention network layer to obtain the multimodal feature of the human body posture.
[0069] Optionally, the posture estimation layer and the one-dimensional convolution layer are connected to the attention network layer, and the attention network layer fuses the position vector and the global pressure intensity features to obtain multimodal features of the human posture.
[0070] In an optional embodiment, the attention network layer adaptively weights the features of different modalities by learning weights, that is, weights are assigned to the position vector and the global pressure intensity feature. The size of the weight reflects the importance of each modality in the current situation, thereby obtaining multimodal features of the human body posture.
[0071] In an optional embodiment, the position vector and the global pressure intensity feature are directly concatenated together to form a comprehensive feature vector to obtain a multimodal feature of the human body posture.
[0072] Step S308: Output the adjustment parameters of each object in the target area according to the multimodal characteristics of the human body posture through the output layer.
[0073] Optionally, the server outputs adjustment parameters of each object in the target area according to the multimodal characteristics of the human body posture through the output layer.
[0074] In actual applications, based on the human joints and pressure characteristics of the target object, height XXcm, weight XXkg, estimated sitting height and elbow height, assuming that the sitting height is XX% of the height, assuming that the elbow height is XX% of the sitting height, feet flat, the pressure is XX pounds, knees bent about XX degrees, the table height is slightly higher than the elbow height, etc., the output smart table height is YY, the smart seat height is YY, the width is YY, the display height is YY, and the lighting angle is YY.
[0075] In actual applications, when the camera detects that the user's shoulders are tilted and the pressure sensor senses that the pressure on one side of the hip is too high, the server can automatically adjust the chair back angle and table height based on this information to make the user sit more comfortably.
[0076] In this embodiment, the multimodal features are fused through the target posture adjustment model, which can simultaneously consider the user's posture and contact pressure, so that the adjustment parameters are more adapted to the target object, thereby improving the accuracy of the adjustment parameters for different target objects.
[0077] In an exemplary embodiment, Figure 4 As shown, the position vector and the global pressure intensity feature are fused through the attention network layer to obtain the multimodal features of the human body posture, including steps S402 to S408. Among them:
[0078] Step S402: for the position vector, generate a first query vector, a first key vector and a first value vector through linear transformation; for the global pressure intensity feature, generate a second query vector, a second key vector and a second value vector through linear transformation.
[0079] Optionally, the server generates a first query vector Q_pose, a first key vector K_pose and a first value vector V_pose for the position vector F_pose through linear transformation; similarly, a second query vector Q_pressure, a second key vector K_pressure and a second value vector V_pressure of the global pressure intensity feature are obtained.
[0080] Step S404: Determine a first correlation score by performing a dot product on the first query vector and the second key vector; determine a second correlation score by performing a dot product on the second query vector and the first key vector.
[0081] Optionally, in order to fuse the position feature and the pressure feature, the server needs to calculate the correlation between these features. The server calculates the dot product of the first query vector Q_pose and the second key vector K_pressure, and obtains the result by Score1(Qpose,Kpressure)=Qpose·Kpressure T , and obtain the first correlation score Score1.
[0082] Similarly, the server calculates the dot product of the second query vector Q_pressure and the first key vector K_pose, through Score1(Qpressure,Kpose) = Qpressure·Kpose T , and obtain the second correlation score Score2.
[0083] Step S406: determining a first weight vector of the position vector and a second weight vector of the global pressure intensity feature according to the first correlation score and the second correlation score.
[0084] Optionally, the server normalizes the scores using a softmax function and calculates the attention weight α1 of the position to pressure as in formula (1).
[0085] α1=Softmax(Score1) Formula (1)
[0086] The attention weight α2 of pressure on posture is calculated as shown in formula (2).
[0087] α2=Softmax(Score2) Formula (2)
[0088] Step S408, determining the multimodal features of the human body posture according to the first value vector, the first weight vector, the second value vector and the second weight vector.
[0089] Optionally, the server fuses the position and pressure features by weighted summation:
[0090] The position feature weight is calculated using formula (3).
[0091] Fpose-fused = α1 ·Vpose formula (3)
[0092] The pressure feature weight is calculated using formula (4).
[0093] Fpressure-fused = α2 ·Vpressure formula (4)
[0094] The weighted features of position and pressure are fused by formula (5) to determine the multimodal feature Ffused of human posture.
[0095] Ffused = [Fpose-fused; Fpressure-fused] Formula (5)
[0096] In this embodiment, the features of the two modalities are fused through the attention mechanism, which can not only dynamically adjust the importance of different modal features in the decision-making process, but also more accurately capture the interaction between the two.
[0097] In an exemplary embodiment, Figure 5 As shown, the position vector and the global pressure intensity feature are fused through the attention network layer to obtain the multimodal features of the human body posture, including steps S502 to S504. Among them:
[0098] Step S502: extracting the characteristics of the contact pressure information to obtain the pressure intensity characteristics at different positions.
[0099] Optionally, the server extracts features of the contact pressure information to obtain pressure intensity features at different locations, such as the pressure intensity features of the target object pressing on the desktop, the pressure intensity features of the target object on a chair, or the pressure intensity features of the target object on the chair with both feet on the ground.
[0100] Step S504, determining the global pressure intensity characteristics of each object based on the pressure intensity characteristics at different positions and the corresponding weights.
[0101] Optionally, the server determines the global pressure intensity characteristics of each object by combining the pressure intensity characteristics of different positions and the corresponding weights, and can assign a greater weight to important positions and a smaller weight to positions that can be ignored.
[0102] In this embodiment, by taking into account the pressure intensity characteristics at different positions and the global pressure characteristic intensity, more accurate adaptation for the target object can be improved.
[0103] In an exemplary embodiment, Figure 6 As shown, before the image information and contact pressure information are input into the trained target posture adjustment model, steps S602 to S604 are also included. Among them:
[0104] Step S602: acquiring sample image information of the sample object and sample contact pressure information of the sample object on an object in a target area.
[0105] Optionally, the server obtains a large number of samples, including the optimal adjustment positions of tables and chairs for different sitting postures. These annotations can be generated by expert advice or an automatic annotation system. Sample image information of sample objects and sample contact pressure information of items in the target area of the sample objects are obtained.
[0106] Step S604: training the initial posture adjustment model using the sample image information and the sample contact pressure information to obtain a target posture adjustment model.
[0107] Optionally, the server trains the initial posture adjustment model through sample image information and sample contact pressure information, and obtains the target posture adjustment model when the loss function converges, wherein the loss function can use the mean square error (MSE) loss function to calculate the error between the adjustment parameters output by the model and the actual annotation value. According to the importance of different modal features, the loss function can be designed in a weighted form. For example, if pressure data is more important in some cases, the weight of pressure-related features is increased accordingly.
[0108] In this embodiment, a multi-modal target posture adjustment model can be obtained by collecting multi-modal sample data.
[0109] In an exemplary embodiment, after inputting image information and contact pressure information into a trained target posture adjustment model and obtaining the adjustment parameters of each object in the target area output by the target posture adjustment model, it also includes: when the target object maintains an unchanged posture within a target time period, outputting voice information; and adjusting the adjustment parameters corresponding to each object, and outputting new adjustment parameters corresponding to each object.
[0110] Optionally, when the target object maintains an unchanged posture within the target time period, a voice message is output to remind the target object to sit for a long time; the server adjusts the adjustment parameters corresponding to each item, such as tilting the chair by XX degrees, and outputs new adjustment parameters corresponding to each item. The server adjusts the position of each item according to the new adjustment parameters corresponding to each item.
[0111] In this embodiment, through voice prompts and parameter adjustments, the target object can be reminded of the need for long periods of time and, together with the tables and chairs, the target object can conduct business in a healthier posture.
[0112] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0113] Based on the same inventive concept, the embodiment of the present application also provides an adjustment device based on a multimodal fusion strategy for implementing the adjustment method based on a multimodal fusion strategy involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more adjustment devices based on a multimodal fusion strategy provided below can refer to the limitations of the adjustment method based on a multimodal fusion strategy above, and will not be repeated here.
[0114] In an exemplary embodiment, Figure 7 As shown, an adjustment device based on a multimodal fusion strategy is provided, comprising: an identification module 701, an acquisition module 702 and a processing module 703, wherein:
[0115] The recognition module 701 is used to recognize a target object in a target area.
[0116] The acquisition module 702 is used to acquire image information of the target object through an image acquisition device; and acquire contact pressure information of the target object on an object in a target area through a pressure sensor.
[0117] The processing module 703 is used to input the image information and contact pressure information into the trained target posture adjustment model, and obtain the adjustment parameters of each object in the target area output by the target posture adjustment model, wherein the target posture adjustment model is trained based on the sample image information and the sample contact pressure information through a multimodal fusion strategy.
[0118] In an exemplary embodiment, the processing module 703 includes:
[0119] The convolution unit is used to extract the multi-level image features of the image information through a multi-dimensional convolution layer to obtain the joint point features of the human body posture; and to extract the features of the contact pressure information through a one-dimensional convolution layer to obtain the global pressure intensity features.
[0120] The position determination unit is used to determine the position vector of the joint point of the human body posture according to the joint point features of the human body posture through the posture estimation layer.
[0121] The fusion unit is used to fuse the position vector and the global pressure intensity features through the attention network layer to obtain the multimodal features of the human body posture;
[0122] The output unit is used to output the adjustment parameters of each object in the target area according to the multimodal characteristics of the human body posture through the output layer.
[0123] In an exemplary embodiment, the fusion unit comprises:
[0124] A vector generation subunit, for generating a first query vector, a first key vector and a first value vector through linear transformation for the position vector; for generating a second query vector, a second key vector and a second value vector through linear transformation for the global pressure intensity feature;
[0125] A correlation score subunit, for determining a first correlation score by performing a dot product on the first query vector and the second key vector; and determining a second correlation score by performing a dot product on the second query vector and the first key vector;
[0126] A weight generating subunit, configured to determine a first weight vector of the position vector and a second weight vector of the global pressure intensity feature according to the first correlation score and the second correlation score;
[0127] The multimodal feature determination subunit is used to determine the multimodal features of the human body posture according to the first value vector, the first weight vector, the second value vector and the second weight vector.
[0128] In an exemplary embodiment, the convolution unit is also used to extract features of contact pressure information to obtain pressure intensity features at different positions; based on the pressure intensity features at different positions and the corresponding weights, the global pressure intensity features of each object are determined.
[0129] In an exemplary embodiment, an adjustment device based on a multimodal fusion strategy further includes:
[0130] The training module is used to obtain sample image information of the sample object and sample contact pressure information of the sample object in the target area; the initial posture adjustment model is trained through the sample image information and the sample contact pressure information to obtain the target posture adjustment model.
[0131] In an exemplary embodiment, an adjustment device based on a multimodal fusion strategy further includes:
[0132] The adjustment output module is used to output voice information when the target object keeps the same posture within the target time period; and adjust the adjustment parameters corresponding to each object and output new adjustment parameters corresponding to each object.
[0133] Each module in the above-mentioned adjustment device based on multimodal fusion strategy can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0134] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store human posture and pressure distribution data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an adjustment method based on a multimodal fusion strategy is implemented.
[0135] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0136] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0137] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0138] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0139] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0140] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0141] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. An adjustment method based on a multimodal fusion strategy, characterized in that: The method comprises: Identify the target object within the target area; Acquire image information of the target object through an image acquisition device; acquire contact pressure information of the target object within a target area through a pressure sensor; The image information and the contact pressure information are input into a trained target posture adjustment model, and the adjustment parameters of each object in the target area output by the target posture adjustment model are obtained, wherein the target posture adjustment model is trained based on sample image information and sample contact pressure information through a multimodal fusion strategy.
2. The method according to claim 1, characterized in that The step of inputting the image information and the contact pressure information into a trained target posture adjustment model, and obtaining adjustment parameters of each object in the target area output by the target posture adjustment model, includes: Extracting multi-level image features of the image information through a multi-dimensional convolution layer to obtain joint point features of the human body posture; extracting features of the contact pressure information through a one-dimensional convolution layer to obtain global pressure intensity features; Determine the position vector of the joint point of the human body posture according to the joint point features of the human body posture through the posture estimation layer; The position vector and the global pressure intensity feature are fused through an attention network layer to obtain a multimodal feature of the human body posture; The output layer outputs adjustment parameters of each object in the target area according to the multimodal features of the human body posture.
3. The method according to claim 2, characterized in that The position vector and the global pressure intensity feature are fused through the attention network layer to obtain multimodal features of human posture, including: For the position vector, generate a first query vector, a first key vector and a first value vector through linear transformation; For the global pressure intensity feature, generate a second query vector, a second key vector and a second value vector through linear transformation; Determining a first relevance score by performing a dot product on the first query vector and the second key vector; Determine a second relevance score by performing a dot product on the second query vector and the first key vector; Determine a first weight vector for the position vector and a second weight vector for the global pressure intensity feature by using the first correlation score and the second correlation score; A multimodal feature of a human body posture is determined according to the first value vector, the first weight vector, the second value vector, and the second weight vector.
4. The method according to claim 1, characterized in that: The step of extracting the contact pressure information features through a one-dimensional convolutional layer to obtain a global pressure intensity feature includes: Extracting features of the contact pressure information to obtain pressure intensity features at different positions; Based on the pressure intensity characteristics at different positions and the corresponding weights, the global pressure intensity characteristics of each object are determined.
5. The method according to claim 1, characterized in that Before inputting the image information and the contact pressure information into the trained target posture adjustment model, the method further includes: Acquiring sample image information of the sample object and sample contact pressure information of the sample object on an object in a target area; The initial posture adjustment model is trained using the sample image information and the sample contact pressure information to obtain a target posture adjustment model.
6. The method according to claim 1, characterized in that After inputting the image information and the contact pressure information into the trained target posture adjustment model and obtaining the adjustment parameters of each object in the target area output by the target posture adjustment model, the method further includes: When the target object keeps the same posture within the target time period, the voice information is output; and the adjustment parameters corresponding to each of the items are adjusted, and new adjustment parameters corresponding to each of the items are output.
7. An adjustment device based on a multimodal fusion strategy, characterized in that: The device comprises: A recognition module, used for identifying a target object in a target area; An acquisition module is used to acquire image information of the target object through an image acquisition device; and acquire contact pressure information of the target object on an object in a target area through a pressure sensor; A processing module is used to input the image information and the contact pressure information into a trained target posture adjustment model, and obtain the adjustment parameters of each object in the target area output by the target posture adjustment model, wherein the target posture adjustment model is trained based on sample image information and sample contact pressure information through a multimodal fusion strategy.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.