Body measurement size prediction method and device fusing multi-modal data and medium

Abstract

The embodiment of the invention discloses a body measurement size prediction method and device fusing multi-modal data and a medium, is applied to the technical field of artificial intelligence, and is used for solving the problem that an existing body measurement mode is low in efficiency and low in accuracy. The method comprises the following steps: training an initial visual feature extraction model based on a sample human body image and a user basis corresponding to the sample human body image to obtain a preset visual feature extraction model; acquiring a multi-angle human body image of a current user and basic parameters of the user, and splicing the multi-angle human body image and the basic parameters of the user to obtain multi-channel input data; performing feature extraction on the multi-channel input data based on a preset visual feature extraction model to obtain corresponding visual features, and inputting the visual features into a preset full-connection layer to obtain feature data corresponding to the current user; and processing the feature data according to a preset auto-encoder model to obtain a prediction result of the current user.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and medium for predicting body size by integrating multimodal data. Background Art

[0002] With the continuous development and popularization of Internet technology, e-commerce has become an indispensable part of people's daily life. Consumers can easily purchase various goods through e-commerce platforms and enjoy a convenient shopping experience. On this basis, personalized customization services are gradually gaining popularity. Consumers are no longer satisfied with the same goods on the market, but hope to customize unique goods according to their preferences and needs. This trend has prompted the continuous innovation and upgrading of body measurement prediction technology to meet consumers' needs for accurate, efficient and personalized shopping experience. However, traditional body measurement methods often rely on manual measurement, which is not only time-consuming and labor-intensive, but also lacks unified measurement standards.

[0003] The current method of automatically determining body size is to make predictions based only on text data entered by the user or only on human body images uploaded by the user. However, the prediction results are affected by the single source of information, which limits the accuracy and robustness of the prediction results. The second method is to make predictions based on computers and their vision. However, the current method of making predictions based on machine vision generally estimates human three-dimensional data based on simple contour extraction and proportion conversion, without considering the differences in body shape among different people. In addition, it is affected by multiple factors such as image quality, lighting conditions, and user posture, resulting in unstable prediction results of body size. Summary of the invention

[0004] In order to solve the above technical problems, one or more embodiments of the present specification provide a method, device and medium for predicting body size by fusing multimodal data.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of the present specification provide a method for predicting body size by fusing multimodal data, the method comprising:

[0007] Based on an existing database, a sample human image and basic parameters of a user corresponding to the sample human image are obtained, so as to train an initial visual feature extraction model based on the sample human image and basic parameters of the user corresponding to the sample human image to obtain a preset visual feature extraction model;

[0008] Acquire a multi-angle human body image and basic parameters of the current user, and splice the multi-angle image and the basic parameters of the user to obtain multi-channel input data; wherein the multi-angle image includes at least two angles of the human body image of the current user;

[0009] Performing feature extraction on the multi-channel input data based on a preset visual feature extraction model to obtain corresponding visual features, and inputting the visual features into a preset fully connected layer to obtain feature data corresponding to the current user;

[0010] The feature data is processed according to a preset autoencoder model to obtain a prediction result for the current user.

[0011] Optionally, in one or more embodiments of the present specification, obtaining a sample human body image and basic parameters of a user corresponding to the sample human body image based on an existing database, training an initial visual feature extraction model based on the sample human body image and the basic parameters of the user corresponding to the sample human body image, and obtaining a preset visual feature extraction model specifically includes:

[0012] Based on an existing database, a sample human body image and basic user parameters corresponding to the sample human body image are obtained, and the sample human body image obtained from the existing database and the basic user parameters corresponding to the sample human body image are spliced ​​to obtain multi-channel input sample data;

[0013] Performing feature extraction on the multi-channel input sample data based on an initial visual feature extraction model to obtain corresponding visual feature samples, and inputting the visual feature samples into a preset fully connected layer to obtain sample feature data corresponding to the sample human body image;

[0014] Processing the feature data according to a preset autoencoder model to obtain a prediction result corresponding to the sample human body image;

[0015] Determining a loss value of the preset visual feature extraction model according to a prediction result corresponding to the sample human body image and an actual measurement result corresponding to the sample human body image;

[0016] According to the loss value corresponding to each measurement part, the mean value corresponding to each measurement part is determined, and the adjustment weight corresponding to each measurement part is determined, so as to iteratively update the initial visual feature extraction model based on the adjustment weight to obtain the trained preset visual feature extraction model.

[0017] Optionally, in one or more embodiments of the present specification, before obtaining a multi-angle human body image and basic user parameters of the current user to splice the multi-angle image and the basic user parameters to obtain multi-channel input data, the method further includes:

[0018] Based on a preset acquisition device, a human body image of the current user is acquired to pre-process the human body image to obtain processed human body images at multiple angles as initial multi-angle human body images of the current user;

[0019] Performing subject detection on the initial multi-angle human body images according to a preset target detection model, locating the target human body in each of the initial multi-angle human body images, and obtaining a current multi-angle human body image with a bounding box; wherein the bounding box is used to mark the target human body;

[0020] Filtering the current multi-angle human body image according to a preset posture detection model to obtain a valid multi-angle human body image;

[0021] Performing background segmentation on the effective multi-angle human image according to a preset semantic segmentation model to obtain a segmented foreground, and constructing a transparent background image based on the size of the effective multi-angle human image;

[0022] Merging the foreground image and the transparent background image to obtain a multi-angle human body image to be converted;

[0023] The format of the multi-angle image to be converted is converted to perform binarization processing on the multi-angle image to be converted according to the value of the alpha channel to obtain a multi-angle human body image of the current user.

[0024] Optionally, in one or more embodiments of the present specification, a human body image of the current user is collected based on a preset collection device to pre-process the human body image to obtain processed human body images at multiple angles as the initial multi-angle human body image of the current user, specifically including:

[0025] Collecting a human body image of the current user based on a preset acquisition device, and converting the human body image into a second color space based on a first color space; wherein the first color space corresponds to machine display and the second color space corresponds to human perception;

[0026] Determine brightness information of the human body image according to the second color space, segment the human body image based on a preset segmentation size, and obtain brightness mean values ​​of each human body image region based on the brightness information;

[0027] Determining a normalization factor of each of the human body image regions based on a ratio of the brightness mean to the preset local brightness reference;

[0028] Transforming the pixel values ​​of the human body image region to be processed based on a power law function to obtain a corrected human body image region to be processed, and adjusting the corrected human body image region to be processed according to a normalization factor of each human body image region to obtain an initial corrected human body image;

[0029] Traversing the pixel values ​​of each point in the initial corrected human body image to determine a global histogram corresponding to the initial corrected human body image based on the number of occurrences of each pixel value, and normalizing the global histogram to obtain a normalized global histogram;

[0030] Calculating the cumulative distribution function of the normalized global histogram to transform the initial corrected human body image according to the cumulative distribution function to obtain a corrected human body image;

[0031] The corrected human body image is converted into the first color space to obtain a processed human body image, so as to determine an initial multi-angle human body image of the current user according to the human body images processed at multiple angles of the current user.

[0032] Optionally, in one or more embodiments of the present specification, performing subject detection on the initial multi-angle human body images according to a preset target detection model and locating the target human body in each of the initial multi-angle human body images specifically includes:

[0033] The initial multi-angle human body images are input into a preset target detection model to obtain a plurality of output results corresponding to each initial multi-angle human body image; wherein the output results include: a bounding box and a category.

[0034] The confidence level of each output result;

[0035] Filtering the multiple output results based on the confidence level of each output result to obtain a candidate output result;

[0036] Determine the center point of the bounding box corresponding to each selected output result and the image center point corresponding to the initial multi-angle human body image, so as to determine the centrality score of each bounding box based on the Euclidean distance between the center point of the bounding box and the image center point;

[0037] Obtaining an area ratio between the area of ​​each of the bounding boxes and the area of ​​the multi-angle human body image corresponding to the bounding box, so as to determine an area score of each bounding box according to the area ratio;

[0038] Determine a weighted value of the centrality score and the area score, screen each of the boundary values ​​based on the weighted value, determine an optimal bounding box, and locate the target human body of the initial multi-angle human body image according to the optimal bounding box.

[0039] Optionally, in one or more embodiments of the present specification, obtaining a multi-angle human body image and basic user parameters of the current user to splice the multi-angle image and the basic user parameters to obtain multi-channel input data specifically includes:

[0040] Filling the multi-angle human body image of the current user according to a preset first size to obtain a filled multi-angle human body image;

[0041] splicing the filled multi-angle human body images based on the preset second size to obtain a spliced ​​multi-angle human body image;

[0042] Standardizing the basic parameters of the user to obtain standard basic parameters of the user; wherein the basic parameters of the user include: height and weight;

[0043] The spliced ​​multi-angle human body images and the user standard basic parameters are distributed to a preset number of channels to obtain multi-channel input data.

[0044] Optionally, in one or more embodiments of the present specification, feature extraction is performed on the multi-channel input data based on a preset visual feature extraction model to obtain corresponding visual features, and the visual features are input into a preset fully connected layer to obtain feature data corresponding to the current user, specifically including:

[0045] Performing data enhancement processing on the multi-channel input samples to obtain enhanced multi-channel input data; wherein the data enhancement processing includes: rotation operation, scaling operation, and flipping operation;

[0046] Inputting the enhanced multi-channel input data into a preset visual feature extraction model to perform feature extraction on the multi-channel input data based on the preset visual feature extraction model to obtain corresponding visual features; wherein the network structure of the preset visual feature extraction model is smaller than that of the preset semantic segmentation model;

[0047] The visual features are input into a fully connected layer consisting of a preset number of hidden layers to obtain feature data corresponding to a preset number of current users.

[0048] Optionally, in one or more embodiments of the present specification, processing the feature data according to a preset autoencoder model to obtain a prediction result of the current user specifically includes:

[0049] Inputting the feature data into the preset autoencoder model, so as to perform a convolution operation on the feature data in an encoder of the preset autoencoder model, so as to compress the feature data to obtain a low-dimensional latent space representation;

[0050] Reconstructing the low-dimensional latent space representation based on a decoder of a preset autoencoder model to obtain reconstructed feature data;

[0051] Determine a reconstruction error according to a difference between the reconstructed feature data and the feature data, so as to determine whether the performance of the preset autoencoder meets the requirements according to the reconstruction error;

[0052] If so, the low-dimensional latent space representation is input into a classifier for prediction to obtain a prediction result for the current user.

[0053] One or more embodiments of the present specification provide a body size prediction device that fuses multimodal data, the device comprising:

[0054] at least one processor; and,

[0055] a memory communicatively connected to the at least one processor; wherein,

[0056] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods.

[0057] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute any of the above-described methods.

[0058] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects:

[0059] By using multi-angle human images, the user's body shape information can be captured more comprehensively, including morphological changes at different angles. The multi-angle human images are spliced ​​with the user's basic parameters to form multi-channel input sample data, so that the model can simultaneously use the intuitive information such as appearance and posture in the image and the quantitative information in the basic parameters. This fusion of multimodal data can capture more comprehensive user body shape information, not only considering the visual morphological differences, but also integrating the user's basic parameters, which helps the model to understand the human body structure more comprehensively, thereby improving the accuracy of the prediction results. During the training process, by comparing the predicted results with the actual measurement results and calculating the loss values ​​corresponding to each measured part, it helps to find the prediction error of the model on different parts, and determine the adjustment weights corresponding to each measured part according to the error size. Based on the customized weight allocation for different parts, the accuracy control of important human parts is achieved, which not only avoids the complex 3D joint projection calculation and fixed-point calculation, but also significantly reduces the difficulty and amount of calculation of data collection, thereby simplifying the training process and improving the calculation efficiency. By using the preset autoencoder model to process the feature data extracted from the multi-channel input data by the preset visual feature extraction model, the potential information and patterns in the feature data can be further mined, thereby improving the accuracy of the prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0061] Figure 1 A schematic diagram of a method for predicting body size by fusing multimodal data provided in an embodiment of this specification;

[0062] Figure 2 A logical schematic diagram of body size prediction by fusing multimodal data provided in an embodiment of this specification;

[0063] Figure 3 A schematic diagram of the structure of a body size prediction device integrating multimodal data provided in an embodiment of this specification;

[0064] Figure 4 A schematic diagram of the structure of a non-volatile storage medium provided in an embodiment of this specification. DETAILED DESCRIPTION

[0065] The embodiments of the present specification provide a method, device and medium for predicting body size by fusing multimodal data.

[0066] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0067] like Figure 1 As shown, the embodiment of this specification provides a flow chart of a method for predicting body size by fusing multimodal data. Figure 1 It can be seen that in one or more embodiments of the present specification, a method for predicting body size by fusing multimodal data includes the following steps:

[0068] S101: acquiring sample human images and basic parameters of users corresponding to the sample human images based on an existing database, training an initial visual feature extraction model based on the sample human images and basic parameters of users corresponding to the sample human images, and obtaining a preset visual feature extraction model.

[0069] In order to improve the accuracy of body size prediction, in the embodiment of this specification, sample human images and basic user parameters corresponding to the sample human images are first obtained based on the existing database. By obtaining the sample human images and basic user parameters corresponding to the sample human images, it helps to avoid the problem that the features extracted by the visual feature extraction model trained with single-dimensional data cannot take into account individual differences. Then, the initial visual feature extraction model is trained with a large number of sample human images and their corresponding basic user parameters, so that the initial visual feature extraction model can learn more accurate and comprehensive feature representations, thereby helping to improve the recognition accuracy of unknown human images in the future.

[0070] Specifically, in one or more embodiments of the present specification, a sample human body image and basic parameters of a user corresponding to the sample human body image are obtained based on an existing database, so as to train an initial visual feature extraction model based on the sample human body image and the basic parameters of the user corresponding to the sample human body image to obtain a preset visual feature extraction model, which specifically includes the following process:

[0071] First, based on the existing database, a sample human body image and the basic parameters of the user corresponding to the sample human body image are obtained, so as to splice the sample human body image obtained from the existing database and the basic parameters of the user corresponding to the sample human body image to obtain multi-channel input sample data. Here, it should be noted that the splicing method here can distribute the sample human body images and basic user parameters of multiple angles to multiple channels, so as to obtain multi-channel input sample data. Then, based on the initial visual feature extraction model, feature extraction is performed on the multi-channel input sample data to obtain the corresponding visual feature samples, and the visual feature samples are input into the preset fully connected layer to obtain the sample feature data corresponding to the sample human body image. Then, the feature data is processed according to the preset autoencoder model to obtain the prediction result corresponding to the sample human body image. According to the prediction result corresponding to the sample human body image and the actual measurement result corresponding to the sample human body image, the loss value of the preset visual feature extraction model is determined. Then, according to the loss value corresponding to each measurement part, the mean corresponding to each measurement part is determined, and the adjustment weight corresponding to each measurement part is determined, so as to iteratively update the initial visual feature extraction model based on the adjustment weight to obtain the trained preset visual feature extraction model.

[0072] In this process, by splicing the sample human body image and the user's basic parameters, two different modal data, to form multi-channel input sample data, the model can simultaneously use the intuitive information such as appearance and posture in the image and the quantitative information in the basic parameters. This fusion of multi-modal data helps the model to understand the human body structure more comprehensively, thereby improving the accuracy of the prediction results. The initial visual feature extraction model is responsible for extracting key features from the multi-channel input data, which are then sent to the preset fully connected layer for further processing and obtaining sample feature data. This process helps to extract and abstract human body features layer by layer, enhancing the robustness of the model. At the same time, the processing of feature data by the preset autoencoder model further improves the expressiveness of the features, provides more accurate information for subsequent predictions, and the preset autoencoder model can make the most of the existing large amount of body size data, thereby helping to improve the accuracy of subsequent predictions. By comparing the prediction results with the actual measurement results and calculating the loss values ​​corresponding to each measurement part, it helps to find the prediction errors of the model at different parts, and determine the adjustment weights corresponding to each measurement part according to the error size. By customizing the weight allocation for different parts, the accuracy control of important human body parts is achieved, especially for three-dimensional measurements that require high precision. This method not only avoids complex 3D joint projection calculations and fixed-point calculations, but also significantly reduces the difficulty and amount of calculations for data collection, thereby simplifying the training process and improving computational efficiency. In addition, this method directly uses a large amount of real body data for model training, which is relatively easy to measure but has extremely high authenticity and accuracy. By directly using these body data, the model can better reflect the distribution and changes of actual human body size, and improve the reliability and practicality of the prediction. This innovative application enables the model to adapt to different measurement needs more efficiently and provide accurate and reliable size prediction results in actual operations. The model performance can also be optimized in a more targeted manner, avoiding the problem of ignoring body shape differences, that is, differences in various parts, in the current traditional method, thereby reducing the prediction error.

[0073] S102: Acquire multi-angle human body images and basic parameters of the current user, and splice the multi-angle images and the basic parameters of the user to obtain multi-channel input data; wherein the multi-angle images at least include human body images of the current user at two angles.

[0074] In order to enhance the feature extraction capability, the embodiment of this specification obtains multi-angle images of the current user and the basic parameters of the user, thereby splicing the multi-angle images and the basic parameters of the user to obtain multi-channel input data. It should be noted that the multi-angle image includes at least two angles of the human body image of the current user, such as a frontal human body image and a side human body image. By fusing the visual image with the numerical parameters, it is convenient for the model to process visual information and numerical information simultaneously in the same data space, thereby enhancing the feature extraction capability. And through the method of multi-channel splicing, the spatial consistency of the numerical parameters and the image information is maintained, the problem of feature loss or information inconsistency is avoided, and the processing flow is simplified, the demand for a complex multi-branch network architecture is reduced, the computational complexity is reduced, and the efficiency is improved.

[0075] Specifically, in one or more embodiments of the present specification, obtaining a multi-angle human body image and basic parameters of the current user, so as to splice the multi-angle image and the basic parameters of the user to obtain multi-channel input data, specifically includes the following process:

[0076] First, fill the multi-angle human body image of the current user according to the preset first size to obtain the filled multi-angle human body image. Then, splice the filled multi-angle human body images based on the preset second size to obtain the spliced ​​multi-angle human body image. Standardize the basic parameters of the user to obtain the standard basic parameters of the user. It should be noted that the basic parameters of the user include: height and weight. Then, the spliced ​​multi-angle human body image and the standard basic parameters of the user are allocated to a preset number of channels to obtain multi-channel input data. For example: in a certain application scenario, the front and side human body images can be kept in the original proportion and filled to a size of 480×640 (width×height). Then, the two filled images, that is, the filled proof human body image and the side human body image, are spliced ​​into a 960×640 image. In addition, other widths and heights can also be selected for filling, and the difference is not large after testing. The basic parameters of the user, such as height and weight, will be standardized according to the maximum value, and the standardized range is 0 to 1. The maximum value can be determined based on the statistical results in the training data, or it can be set to a fixed value (for example, height 255cm, weight 255kg). Gender is treated as unknown with 0, and male or female is defined as 1 and 2 respectively, and is also standardized. Subsequently, the standardized data is converted to integer values ​​between 0 and 255 through linear mapping, and an 8-bit depth image with a size of 480×640 is generated. The stitched image includes three channels: front and side views, height, weight and gender information. It should also be noted that the stitching method is not limited to the above. Height and weight can also be used as a single channel, and then other data can be merged into one channel. After testing, the effect is not much different.

[0077] In the above process, by acquiring human body images from multiple angles, the user's body characteristics can be fully captured, including multiple perspectives such as the front, side, and back, which helps to more accurately reflect the user's body details and provide more comprehensive data support for subsequent analysis, modeling, or personalized recommendations. Standardizing the data of basic user parameters such as height and weight can eliminate the differences between different units or dimensions, so that the data of different users can be compared and analyzed under a unified framework, improving the consistency and accuracy of data processing. By splicing the sample human body image and the user's basic parameters, two different modal data, to form multi-channel input sample data, the model can simultaneously use the intuitive information such as appearance and posture in the image and the quantitative information in the basic parameters. This fusion of multimodal data helps the model understand the human body structure more comprehensively, thereby improving the accuracy of the prediction results. The spliced ​​multi-channel input allows the model to capture the potential correlation between different data types during the feature extraction process, improving the prediction accuracy.

[0078] Furthermore, in one or more embodiments of the present specification, before obtaining the multi-angle human body image and the basic parameters of the user of the current user to splice the multi-angle image and the basic parameters of the user to obtain the multi-channel input data, the method further includes the following process:

[0079] In order to extract the human body contour related to the measured body size, and thus achieve targeted analysis, it is necessary to ensure the integrity of the human body contour and the clarity of the edge. Therefore, first, based on the preset acquisition device, the human body image of the current user is collected, so that the human body image is preprocessed, and the processed human body images at multiple angles are obtained as the initial multi-angle human body image of the current user. Then, the initial multi-angle human body image is subjected to subject detection according to the preset target detection model, so as to locate the target human body of the initial multi-angle human body image through the result of the subject detection, and obtain the current multi-angle human body image with a bounding box; wherein it can be understood that the bounding box is used to mark the target human body. The current multi-angle human body image is filtered according to the preset posture detection model to obtain a valid multi-angle human body image. Then, the background of the valid multi-angle human body image is segmented according to the preset semantic segmentation model to obtain the segmented foreground, and a transparent background image is constructed based on the size of the valid multi-angle human body image. Then, the foreground and transparent background images are merged to obtain the multi-angle human body image to be converted. Then, the multi-angle image to be converted is format converted to realize binarization processing of the multi-angle image to be converted according to the value of the alpha channel, and obtain the multi-angle human body image of the current user. For example, DETR or other target detection models will be used in a certain application scenario for subject detection. By comprehensively calculating the centrality and frame size of the target in the image, the main target human body is accurately located, and then posture detection models such as OpenPose or HRNet are used to pre-screen the images to exclude those that do not meet the requirements, such as photos with abnormal postures or partially occluded human bodies. Semantic segmentation models such as U-Net or DeepLab are used for background segmentation to separate the target human body from the complex background, generate a 32-bit PNG image with a transparent background for the main body, and then convert it into a binary multi-angle human image based on whether the alpha channel is transparent.

[0080] In this process, a series of preprocessing steps, including subject detection, posture detection, semantic segmentation, and background replacement, can significantly improve the data quality of human images. The integrity of the human body contour and the clarity of the edges are ensured, providing a high-quality data basis for subsequent body size analysis. Since the background and other interference factors are removed during the preprocessing process, only the clear outline of the target human body is retained, so the human body size information can be extracted more accurately. This helps to reduce measurement errors caused by poor image quality and improve the accuracy of analysis. In addition, through format conversion and binarization processing, the multi-angle images to be converted are converted into a format suitable for subsequent processing. This format conversion not only simplifies the image content, but also facilitates integration and interaction with other image processing software and algorithms. At the same time, the binarization processing further enhances the contrast and clarity of the image, making the human body contour more distinct.

[0081] Furthermore, since the environments of the current users are different, and factors such as lighting may lead to low quality of the human body image, in order to improve the quality of the human body image and thus improve the accuracy of the subsequent detection and recognition process, in one or more embodiments of the present specification, the human body image of the current user is acquired based on a preset acquisition device to pre-process the human body image to obtain processed human body images at multiple angles as the initial multi-angle human body image of the current user, specifically including the following process:

[0082] First, a human body image of the current user is collected based on a preset collection device, and then the human body image is converted to a second color space based on a first color space; wherein the first color space corresponds to the machine display, i.e., the RGB color space, and the second color space corresponds to the human perception, i.e., the HSV color space. Then, the brightness information of the human body image is determined based on the second color space, so that the human body image is segmented based on a preset division size, and the brightness mean of each human body image area is obtained based on the brightness information. Then, based on the ratio of the brightness mean to the preset local brightness reference, the normalization factor of each human body image area is determined. Then, the pixel values ​​of the human body image area to be processed are transformed based on a power law function to obtain a corrected human body image area to be processed, and the corrected human body image area to be processed is adjusted according to the normalization factor of each human body image area to obtain an initial corrected human body image. Then, the pixel values ​​of each point in the initial corrected human body image are traversed, so as to determine the global histogram corresponding to the initial corrected human body image according to the number of occurrences of each pixel value, and the global histogram is normalized to obtain a normalized global histogram. The cumulative distribution function of the normalized global histogram is calculated to transform the initial corrected human body image according to the cumulative distribution function to obtain the corrected human body image. The corrected human body image is converted into a first color space to obtain a processed human body image, so as to determine the initial multi-angle human body image of the current user according to the human body images processed at multiple angles of the current user.

[0083] In the image enhancement process, by converting the human body image from the first color space corresponding to the machine display to the second color space corresponding to human perception, the authenticity of the color in the image and the perceptual characteristics of human vision can be more accurately reflected. This conversion helps to reduce the color distortion problem caused by color space mismatch and improve the color restoration of the image. Then, based on the brightness information determined in the second color space, the human body image is segmented and the brightness mean of each area is calculated, and then the pixel value is transformed and adjusted by the normalization factor and the power law function. This process can effectively correct the uneven brightness problem in the image, make the overall brightness distribution of the image more uniform, and improve the clarity and visibility of the image. Global histogram equalization can be achieved by normalizing the global histogram of the initial corrected human body image and calculating the cumulative distribution function, and then transforming the image. Through the dynamic local adjustment process of the human body image area and the global adjustment process of the global histogram, the consistency and coherence of the global range are achieved while ensuring the local image quality. After completing the global histogram equalization, the corrected human body image is converted back to the first color space to ensure the overall balance and coordination of the image in subsequent processing, which helps to improve the accuracy of subsequent processing.

[0084] Further, in one or more embodiments of the present specification, subject detection is performed on the initial multi-angle human images according to a preset target detection model, and the target human body of each initial multi-angle human body image is located, which specifically includes the following process:

[0085] The initial multi-angle human image is input into the preset target detection model, so as to obtain multiple output results corresponding to each initial multi-angle human image. Among them, the output results include: bounding box, category. The confidence of each output result. Then, the multiple output results are filtered according to the confidence of each output result to obtain the output results to be selected. The center point of the bounding box corresponding to each output result to be selected and the center point of the image corresponding to the initial multi-angle human image are determined, and the centrality score of each bounding box is determined based on the Euclidean distance between the center point of the bounding box and the center point of the image. At the same time, the area ratio between the area of ​​each bounding box and the area of ​​the multi-angle human image corresponding to the bounding box is obtained to determine the area score of each bounding box according to the area ratio, and then the centrality score and the area score are combined, that is, the weighted value of the centrality score and the area score is determined, so as to screen each boundary value according to the weighted value, determine the best bounding box, and locate the target human body of the initial multi-angle human image according to the best bounding box.

[0086] Specifically, in a certain application scenario, the initial multi-angle human body image is input into a preset target detection model, and the predicted bounding boxes, categories, and confidence levels are output. For example, when the confidence threshold is set to 0.9, it is necessary to ensure that only the detection results with high confidence are retained. When screening and scoring the bounding boxes, calculate the Euclidean distance between the center point of each bounding box and the center of the image, and at the same time calculate the ratio of the area of the bounding box to the total area of the image as the area score. Combine the centrality score and the area score, and combine the weights to obtain the final score of each bounding box, and select the best bounding box according to the score. For example, the weight ratio of centrality to area scoring is 1:2. That is, in a certain application scenario, the confidence threshold is set to 0.9, which means that we will only retain those bounding boxes with a confidence level greater than or equal to 0.9. Suppose the model outputs the following three bounding boxes: Bounding box A: confidence level 0.95; Bounding box B: confidence level 0.85; Bounding box C: confidence level 0.92. Since bounding box B is lower than the preset confidence level, bounding boxes A and C are retained for subsequent processing. For each retained bounding box, calculate the Euclidean distance between its center point and the center point of the image. The closer the distance, the higher the centrality score. Suppose: the distance between the center point of bounding box A and the center of the image is d1, and the distance between the center point of bounding box C and the center of the image is d2. When d1 < d2, the centrality score of bounding box A is higher than that of bounding box C. At the same time, calculate the ratio of the area of the bounding box to the total area of the image as the area score. The larger the ratio, the higher the area score. For example: the ratio of the area of bounding box A to the total area of the image is r1, and the ratio of the area of bounding box C to the total area of the image is r2; then assume that both r1 and r2 are within a reasonable range, but r1 is closer to the optimal ratio, so the area score of bounding box A is also higher than that of bounding box C. Set the weight ratio of the centrality score to the area score to 1:2. The comprehensive score is calculated as follows: The comprehensive score of bounding box A = centrality score * 1 + area score * 2; The comprehensive score of bounding box C = centrality score * 1 + area score * 2, and then select the best bounding box according to the result of the comprehensive score.

[0087] In the above screening process, through the preset target detection model, the target human body in the image can be accurately identified, and multiple results including bounding boxes, categories, and confidence levels are output. Subsequently, based on the comprehensive evaluation of confidence filtering, centrality score, and area score, the best bounding box can be further screened out, so as to achieve high-precision positioning of the target human body. This multi-step screening and evaluation mechanism ensures the accuracy and reliability of the detection results. Moreover, by comprehensively considering multiple factors such as the centrality, area, and confidence level of the bounding box, even when there are adverse factors such as noise, blur, or occlusion in the image, the target human body can be accurately located, showing strong robustness. In addition, this process can complete the positioning and recognition of the target human body without manual intervention. This greatly improves the processing efficiency, reduces the labor cost, and reduces the errors that may be brought by human factors.

[0088] S103: Perform feature extraction on the multi-channel input data based on a preset visual feature extraction model to obtain corresponding visual features, and input the visual features into a preset fully connected layer to obtain feature data corresponding to the current user.

[0089] After obtaining the multi-channel input data of the current user based on the above step S102, feature extraction is performed on the multi-channel input data according to the preset visual feature extraction model to obtain the corresponding visual features, and then the visual features are input into the fully connected layer to efficiently process the multi-channel input data and quickly output the user feature data. This method can not only effectively reduce the computational complexity of the model, but also ensure that the processing speed is improved while maintaining accuracy.

[0090] Specifically, in one or more embodiments of the present specification, feature extraction is performed on multi-channel input data based on a preset visual feature extraction model to obtain corresponding visual features, and the visual features are input into a preset fully connected layer to obtain feature data corresponding to the current user, specifically including the following process:

[0091] First, data enhancement processing is performed on the multi-channel input samples to obtain enhanced multi-channel input data; it should be noted that the data enhancement processing includes: rotation operation, scaling operation, and flipping operation. Then the enhanced multi-channel input data is input into the preset visual feature extraction model, so that the multi-channel input data is feature extracted according to the preset visual feature extraction model to obtain the corresponding visual features; since the previous preprocessing steps such as background segmentation and binarization processing are already very perfect, the extracted contour map information is highly concise, so a smaller network structure can be used to extract enough information from it, so the preset semantic segmentation model preset visual feature extraction model network structure is smaller than the preset semantic segmentation model. Then the visual features are input into the fully connected layer composed of a preset number of hidden layers, so as to obtain the feature data corresponding to the preset number of current users. Through this process, not only the computational complexity of the model can be effectively reduced, but also the processing speed can be improved while maintaining accuracy.

[0092] S104: Processing the feature data according to a preset autoencoder model to obtain a prediction result for the current user.

[0093] like Figure 2 As shown, after obtaining the feature data corresponding to the current user, the feature data is processed according to the preset autoencoder model in the embodiment of this specification to obtain the prediction result of the current user. By using the preset autoencoder model to process the feature data, the potential information and patterns in the feature data can be further mined, thereby improving the accuracy of the prediction result.

[0094] Specifically, in one or more embodiments of the present specification, the feature data is processed according to the preset autoencoder model to obtain the prediction result of the current user, which specifically includes the following process:

[0095] The feature data is input into the preset autoencoder model, so that the encoder of the preset autoencoder model performs convolution operation on the feature data, and the feature data is compressed to obtain a low-dimensional latent space representation. Then the low-dimensional latent space representation is reconstructed based on the decoder of the preset autoencoder model to obtain the reconstructed feature data. According to the difference between the reconstructed feature data and the feature data, the reconstruction error is determined, and according to the reconstruction error, it is determined whether the performance of the preset autoencoder meets the requirements. If it meets the requirements, the low-dimensional latent space representation is input into the classifier for prediction to obtain the prediction result of the current user. The preset autoencoder can automatically learn the feature representation from the input data without manually designing the features. This not only saves labor costs, but also avoids the subjectivity and bias of human factors in feature design. At the same time, through self-learning, the autoencoder can continuously optimize the feature representation and improve the prediction accuracy. By processing the feature data through the autoencoder, a more compact, effective and representative feature representation can be obtained. These feature representations can better reflect the essential characteristics of the data, thereby improving the accuracy of the prediction.

[0096] That is, in the embodiment of this specification, the encoder is preset to capture the spatial correlation between human body parts, which consists of an encoder and a decoder. The encoder part gradually extracts high-order features of the input data through a series of 1D convolutional layers, and each convolutional layer is followed by a BatchNormalization layer and a ReLU activation function. After the convolution operation, the feature map is converted into an input suitable for the fully connected layer through a flattening operation. Finally, after the fully connected layer, the encoder compresses the input features into a low-dimensional latent space representation. The decoder part reconstructs the latent representation into a high-dimensional feature output through a deconvolution operation. Corresponding to the encoder, the decoder also includes activation functions such as BatchNormalization layers and ReLU to ensure the stability and expressiveness of the output. Finally, the decoder restores the information in the latent space to feature data of the same dimension as the original input. Compared with the end-to-end direct prediction method, the autoencoder compresses and retains key features in the latent space, avoiding the information loss problem that may be caused by a simple regression model. Through the step-by-step reconstruction process of the decoder, the autoencoder can capture the deep correlation between the input features in a more detailed manner, thereby providing a more accurate size prediction. Moreover, the autoencoder can flexibly handle different types of input data and can adapt to various changes in human body shape. This flexibility enables the autoencoder to maintain high prediction accuracy when facing diverse data inputs.

[0097] like Figure 3As shown, the embodiment of this specification provides a schematic diagram of the structure of a body size prediction device that integrates multimodal data. Figure 3 It can be seen that in one or more embodiments of the present specification, a body size prediction device integrating multimodal data includes:

[0098] at least one processor; and,

[0099] a memory communicatively connected to the at least one processor; wherein,

[0100] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned methods.

[0101] like Figure 4 As shown, the present specification provides a schematic diagram of a non-volatile storage medium structure. Figure 4 It can be seen that in one or more embodiments of the present specification, a non-volatile storage medium stores computer executable instructions 401, and the computer executable instructions 401 can: execute any of the above-mentioned methods.

[0102] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0103] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0104] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A method for predicting body size by fusing multimodal data, characterized in that: The method comprises: Based on an existing database, a sample human image and basic parameters of a user corresponding to the sample human image are obtained, so as to train an initial visual feature extraction model based on the sample human image and basic parameters of the user corresponding to the sample human image to obtain a preset visual feature extraction model; Acquire a multi-angle human body image and basic parameters of the current user, and splice the multi-angle image and the basic parameters of the user to obtain multi-channel input data; wherein the multi-angle image includes at least two angles of the human body image of the current user; Performing feature extraction on the multi-channel input data based on a preset visual feature extraction model to obtain corresponding visual features, and inputting the visual features into a preset fully connected layer to obtain feature data corresponding to the current user; The feature data is processed according to a preset autoencoder model to obtain a prediction result for the current user.

2. The method for body size prediction by fusing multimodal data according to claim 1, characterized in that: Based on an existing database, a sample human image and basic parameters of a user corresponding to the sample human image are obtained, and an initial visual feature extraction model is trained based on the sample human image and the basic parameters of the user corresponding to the sample human image to obtain a preset visual feature extraction model, specifically including: Based on an existing database, a sample human body image and basic user parameters corresponding to the sample human body image are obtained, and the sample human body image obtained from the existing database and the basic user parameters corresponding to the sample human body image are spliced ​​to obtain multi-channel input sample data; Performing feature extraction on the multi-channel input sample data based on an initial visual feature extraction model to obtain corresponding visual feature samples, and inputting the visual feature samples into a preset fully connected layer to obtain sample feature data corresponding to the sample human body image; Processing the feature data according to a preset autoencoder model to obtain a prediction result corresponding to the sample human body image; Determining a loss value of the preset visual feature extraction model according to a prediction result corresponding to the sample human body image and an actual measurement result corresponding to the sample human body image; According to the loss value corresponding to each measurement part, the mean value corresponding to each measurement part is determined, and the adjustment weight corresponding to each measurement part is determined, so as to iteratively update the initial visual feature extraction model based on the adjustment weight to obtain the trained preset visual feature extraction model.

3. The method for body size prediction by fusing multimodal data according to claim 1, characterized in that: Before obtaining a multi-angle human body image and basic parameters of the current user to splice the multi-angle image and the basic parameters of the user to obtain multi-channel input data, the method further includes: Based on a preset acquisition device, a human body image of the current user is acquired to pre-process the human body image to obtain processed human body images at multiple angles as initial multi-angle human body images of the current user; Performing subject detection on the initial multi-angle human body images according to a preset target detection model, locating the target human body in each of the initial multi-angle human body images, and obtaining a current multi-angle human body image with a bounding box; wherein the bounding box is used to mark the target human body; Filtering the current multi-angle human body image according to a preset posture detection model to obtain a valid multi-angle human body image; Performing background segmentation on the effective multi-angle human image according to a preset semantic segmentation model to obtain a segmented foreground, and constructing a transparent background image based on the size of the effective multi-angle human image; Merging the foreground image and the transparent background image to obtain a multi-angle human body image to be converted; The format of the multi-angle image to be converted is converted to perform binarization processing on the multi-angle image to be converted according to the value of the alpha channel to obtain a multi-angle human body image of the current user.

4. The method for body size prediction by fusing multimodal data according to claim 3, characterized in that: The human body image of the current user is collected based on a preset collection device to pre-process the human body image to obtain processed human body images at multiple angles as the initial multi-angle human body image of the current user, specifically including: Collecting a human body image of the current user based on a preset acquisition device, and converting the human body image into a second color space based on a first color space; wherein the first color space corresponds to machine display and the second color space corresponds to human perception; Determine brightness information of the human body image according to the second color space, segment the human body image based on a preset segmentation size, and obtain brightness mean values ​​of each human body image region based on the brightness information; Determining a normalization factor of each of the human body image regions based on a ratio of the brightness mean to the preset local brightness reference; Transforming the pixel values ​​of the human body image region to be processed based on a power law function to obtain a corrected human body image region to be processed, and adjusting the corrected human body image region to be processed according to a normalization factor of each human body image region to obtain an initial corrected human body image; Traversing the pixel values ​​of each point in the initial corrected human body image to determine a global histogram corresponding to the initial corrected human body image based on the number of occurrences of each pixel value, and normalizing the global histogram to obtain a normalized global histogram; Calculating the cumulative distribution function of the normalized global histogram to transform the initial corrected human body image according to the cumulative distribution function to obtain a corrected human body image; The corrected human body image is converted into the first color space to obtain a processed human body image, so as to determine an initial multi-angle human body image of the current user according to the human body images processed at multiple angles of the current user.

5. The method for body size prediction by fusing multimodal data according to claim 3, characterized in that: The performing subject detection on the initial multi-angle human body images according to the preset target detection model and locating the target human body in each of the initial multi-angle human body images specifically includes: Input the initial multi-angle human image into a preset target detection model to obtain multiple output results corresponding to each initial multi-angle human image; wherein the output results include: a bounding box, a category. The confidence level of each output result; Filtering the multiple output results based on the confidence level of each output result to obtain a candidate output result; Determine the center point of the bounding box corresponding to each selected output result and the image center point corresponding to the initial multi-angle human body image, so as to determine the centrality score of each bounding box based on the Euclidean distance between the center point of the bounding box and the image center point; Obtaining an area ratio between the area of ​​each of the bounding boxes and the area of ​​the multi-angle human body image corresponding to the bounding box, so as to determine an area score of each bounding box according to the area ratio; Determine a weighted value of the centrality score and the area score, screen each of the boundary values ​​based on the weighted value, determine an optimal bounding box, and locate the target human body of the initial multi-angle human body image according to the optimal bounding box.

6. The method for body size prediction by fusing multimodal data according to claim 1, characterized in that: Acquiring multi-angle human body images and basic user parameters of the current user, and splicing the multi-angle images and the basic user parameters to obtain multi-channel input data, specifically including: Filling the multi-angle human body image of the current user according to a preset first size to obtain a filled multi-angle human body image; splicing the filled multi-angle human body images based on the preset second size to obtain a spliced ​​multi-angle human body image; Standardizing the basic parameters of the user to obtain standard basic parameters of the user; wherein the basic parameters of the user include: height and weight; The spliced ​​multi-angle human body images and the user standard basic parameters are distributed to a preset number of channels to obtain multi-channel input data.

7. The method for body size prediction by fusing multimodal data according to claim 3, characterized in that: Extracting features from the multi-channel input data based on a preset visual feature extraction model to obtain corresponding visual features, and inputting the visual features into a preset fully connected layer to obtain feature data corresponding to the current user, specifically including: Performing data enhancement processing on the multi-channel input samples to obtain enhanced multi-channel input data; wherein the data enhancement processing includes: rotation operation, scaling operation, and flipping operation; Inputting the enhanced multi-channel input data into a preset visual feature extraction model to perform feature extraction on the multi-channel input data based on the preset visual feature extraction model to obtain corresponding visual features; wherein the network structure of the preset visual feature extraction model is smaller than that of the preset semantic segmentation model; The visual features are input into a fully connected layer consisting of a preset number of hidden layers to obtain feature data corresponding to a preset number of current users.

8. The method for body size prediction by fusing multimodal data according to claim 1, characterized in that: Processing the feature data according to a preset autoencoder model to obtain a prediction result of the current user specifically includes: Inputting the feature data into the preset autoencoder model, so as to perform a convolution operation on the feature data in an encoder of the preset autoencoder model, so as to compress the feature data to obtain a low-dimensional latent space representation; Reconstructing the low-dimensional latent space representation based on a decoder of a preset autoencoder model to obtain reconstructed feature data; Determine a reconstruction error according to a difference between the reconstructed feature data and the feature data, so as to determine whether the performance of the preset autoencoder meets the requirements according to the reconstruction error; If so, the low-dimensional latent space representation is input into a classifier for prediction to obtain a prediction result for the current user.

9. A body size prediction device integrating multimodal data, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: execute any of the methods described in claims 1-8.

10. A non-volatile storage medium storing computer executable instructions, characterized in that: The computer executable instructions can: execute the method described in any one of claims 1 to 8.

Images