Sole evaluation method and system based on large model

Abstract

The invention relates to the technical field of foot evaluation, in particular to a sole evaluation method and system based on a large model. The planta pedis evaluation method comprises the following steps: acquiring first footprint image data; the first footprint image data is standard footprint image data; constructing a training set by using the first footprint image data to finely adjust the open-source image large language model to obtain a footprint evaluation reasoning model; acquiring second footprint image data; the second footprint image data is footprint image data obtained by carrying out modulus taking on a template by a foot of the user; and inputting the second footprint image data into the footprint evaluation reasoning model to obtain a sole evaluation result. According to the method, the pre-trained image language model is finely adjusted to obtain the footprint evaluation reasoning model capable of executing the foot type evaluation task. After a user uploads a real footprint image, the user can obtain information such as the foot length, the foot width, the foot arch type and shoe selection suggestions, the model can be finely adjusted again according to a feedback text sent by the user, and lifelong learning is achieved.

Description

Technical Field

[0001] This application relates to the field of foot assessment technology, and in particular to a foot plantar assessment method and system based on a large model. Background Technology

[0002] In fields such as foot medical rehabilitation, sports science, and personalized insole customization, it is necessary to collect user foot data, perform feature analysis on user feet, and identify user arch characteristics, such as high arch, normal arch, and flat feet.

[0003] Existing foot assessment technologies mainly include foot pressure imaging systems based on pressure sensor arrays, foot morphology estimation methods based on two-dimensional image recognition, and foot modeling methods based on three-dimensional laser scanning.

[0004] Foot pressure imaging systems based on pressure sensor arrays are widely used in hospitals and professional sports rehabilitation facilities. They utilize a high-density pressure sensor array to capture plantar pressure maps during footsteps and then use rule-based software to determine arch shape and gait. However, foot pressure imaging systems are highly sensitive to environmental factors and are easily affected by temperature, humidity, and footstep posture, resulting in data drift issues.

[0005] Foot modeling methods based on 3D laser scanning rely on 3D laser scanning equipment, which is expensive.

[0006] To reduce the cost of intelligent assessment of abnormal foot arch types, this application provides a foot plantar assessment method and a foot plantar assessment system based on a large model. Summary of the Invention

[0007] To overcome the problems existing in related technologies, the first aspect of this application provides a foot evaluation method based on a large model, including: Collect first footprint image data; the first footprint image data includes N footprint images of foot molds; the footprint images of foot molds are footprint images obtained by taking a standard foot mold from a template, and N is an integer greater than or equal to 1; The first footprint image data was used to construct a training set to fine-tune the open-source image language model, resulting in a footprint evaluation inference model. Acquire second footprint image data; the second footprint image data includes a real footprint image; the real footprint image is a footprint image obtained by taking a mold of the user's foot on a template. The second footprint image data is input into the footprint evaluation inference model to obtain the sole evaluation result; the sole evaluation result includes foot length, foot width, arch type and shoe type recommendation text; The footprint evaluation reasoning model is fine-tuned based on the feedback text sent by the user.

[0008] In one implementation, the acquisition of the first footprint image data specifically includes: Place the standard foot mold on the template to take the mold; Take a picture of the footprint on the template to obtain the footprint image of the foot model; The foot mold and footprint images are preprocessed; The footprint image is annotated to obtain the first footprint image data.

[0009] In one embodiment, preprocessing the footprint image specifically includes: An affine transformation is performed on the foot mold footprint image with the direction perpendicular to the plate surface of the template as the frontal viewing direction; The footprint image is subjected to grayscale processing and region segmentation to obtain the footprint image that retains the footprint area.

[0010] In one implementation, a training set is constructed using the first footprint image data to fine-tune an open-source image-based large language model, resulting in a footprint evaluation inference model, specifically including: The first footprint image data is divided into a training set and a validation set; Set LoRA; A first prompt vector is determined based on the prompt word template and the first footprint image data; the prompt word template includes data type prompts, foot length and width prediction prompts, arch type classification prompts, and shoe type recommendation prompts; the training set is loaded, the loss between the prediction results and annotations of the pre-trained large model is calculated, and the process is iterated; The footprint evaluation inference model is obtained by exporting the finely tuned pre-trained large model.

[0011] In one embodiment, the first footprint image data is used to construct a footprint image training set, wherein the foot model footprint image includes an arch region label and a foot length label.

[0012] In one implementation, the second footprint image data is input into the footprint evaluation inference model to obtain the plantar evaluation result, specifically including: obtaining the user ID; Load the real footprint image from the user ID history cache; Add the historical cached real footprint images to the second footprint image data; A second prompt vector is determined based on the prompt word template and the second footprint image data; the second prompt vector is input into the footprint evaluation reasoning model to obtain the plantar evaluation result.

[0013] In one implementation, the footprint evaluation inference model is fine-tuned based on feedback text sent by the user, specifically including: Get the current feedback text for the user ID; The reverse fine-tuning mechanism is determined based on the newly added batch of real footprint images cached by the user ID. If the newly added batch of real footprint images is less than or equal to a preset batch threshold, the attention weight redistribution mechanism is used to optimize the model's inference ability. If the newly added batch of real footprint images is greater than the preset batch threshold, the LoRA mechanism is used to fine-tune the plantar evaluation inference model.

[0014] The second aspect of this application provides a plantar assessment system based on a large model, used to implement the plantar assessment method described in the first aspect of this application.

[0015] The technical solution provided in this application may include the following beneficial effects: In the fine-tuning phase of the footprint evaluation inference model, the first footprint image data is input into a pre-trained image language model for training, enabling the footprint evaluation inference model to perform foot type classification tasks. In the deployment phase of the footprint evaluation inference model, the second footprint image data is input into the model to predict the foot assessment results. This application obtains a footprint evaluation inference model capable of performing foot type assessment tasks through fine-tuning of a pre-trained image language model. After taking a picture of a foot model to obtain a real footprint image, the user inputs it into the footprint evaluation inference model to obtain the foot assessment results, outputting the user's foot length, foot width, arch type, and shoe selection suggestions.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0018] Figure 1 This is a flowchart illustrating the plantar evaluation method based on a large model, as shown in an embodiment of this application.

[0019] Figure 2 This is a schematic diagram of the logical structure of the plantar evaluation system based on a large model, as shown in an embodiment of this application. Detailed Implementation

[0020] Preferred embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0021] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0023] Example 1 Foot assessments of children during their growth and development, as well as the elderly, can promptly identify abnormalities such as flat feet and high arches. However, foot assessments require collecting the user's foot data, and existing technologies based on pressure sensor arrays and 3D laser scanning suffer from excessively high costs.

[0024] To reduce the cost of foot data collection, this application provides a foot evaluation method based on a large model, such as... Figure 1 As shown, it includes the following steps: S100, Acquire the first footprint image data; Specifically, the first footprint image data is the footprint image of a standard foot mold on a template.

[0025] Furthermore, step S100 specifically includes: S101. Place the standard foot mold on the mold-taking template to take the mold; S102. Take a picture of the footprint on the template to obtain the footprint image of the foot model; S103. Preprocess the foot mold and footprint image; S104. The footprint image is annotated to obtain the first footprint image data.

[0026] In step S101, during the molding process, a standard foot mold is pressed onto the molding template, leaving a footprint on the template. The footprint is then captured by an image capturing device to obtain a foot mold image. First foot mold image data is collected. Standard foot molds include normal arch foot molds, high arch foot molds, and flat foot molds.

[0027] In this embodiment, the first footprint image data is used to construct an image training set for fine-tuning an open-source visual large language model, such as Qwen2.5-VL. Qwen2.5-VL supports both text and images as input.

[0028] Step S103 specifically includes the following steps: S1031. Perform an affine transformation on the foot mold footprint image with the direction perpendicular to the plate surface of the template as the frontal viewing direction; S1032. Perform grayscale processing and region segmentation on the foot mold and footprint image to obtain the foot mold and footprint image that retains the footprint region; Furthermore, in step S104, the arch region and foot length in the foot mold and footprint image are labeled. In this embodiment, the labeled foot mold and footprint image is stored and used as a dataset for fine-tuning the large model, enabling the large model to adapt to the foot type classification task.

[0029] S200: Using the first footprint image data, construct a training set to fine-tune the open-source image language model and obtain the footprint evaluation inference model. Step S200 specifically includes: S201. Divide the first footprint image data into a training set and a validation set; S202, Set LoRA; S203. Determine a first prompt vector based on the prompt word template and the first footprint image data; the prompt word template includes data type prompts, foot length and width prediction prompts, arch type classification prompts, and shoe type recommendation prompts. S204. Load the training set, calculate the loss between the prediction results and the labels of the pre-trained large model, and perform iterations. S205. Export the fine-tuned pre-trained large model to obtain the footprint evaluation inference model.

[0030] Step 203 constructs a first prompt vector from the first footprint image data sent by the user using a prompt word template. The first prompt vector is used as input to a pre-trained large model to obtain a specific output and is fine-tuned based on LoRA.

[0031] Furthermore, the prompt word template generates data type prompts, foot length and width prediction prompts, arch type classification prompts, and shoe type recommendation prompts. The data type prompts are used to label the first footprint image data as foot model data. The pre-trained large model outputs foot width values, foot length values, arch type, and shoe type recommendation text based on the foot length and width prediction prompts, arch type classification prompts, and shoe type recommendation prompts.

[0032] In this embodiment, the open-source image large model Qwen-VL and other basic models are used as the backbone structure, and the LoRA (Low-Rank Adaptation) mechanism is used for parameter fine-tuning. During fine-tuning, the collected standard foot mold data is used as the basic training set, and the prompt is marked as "standard foot mold footprint" to guide the model to learn the semantic features related to foot length and arch in the footprint.

[0033] During fine-tuning, the output of the pre-trained large model includes predicted foot length and width values ​​(regression task) and a classification of arch type (classification task), which includes normal arch, flat arch, and high arch. During training, the losses for the regression and classification tasks are calculated separately to optimize the weights of the pre-trained large model.

[0034] After obtaining the footprint evaluation reasoning model, foot evaluation is achieved through the footprint evaluation reasoning model.

[0035] S300, Acquire second footprint image data; Specifically, the second footprint image data includes a real footprint image; the real footprint image is a footprint image obtained by taking a mold of the user's foot on a template.

[0036] In step S300, the real footprint image is subjected to preprocessing steps as described in steps S101 to S104 to obtain the second footprint image data.

[0037] S400: Input the second footprint image data into the footprint evaluation reasoning model to obtain the plantar evaluation result.

[0038] Furthermore, step S400 specifically includes: S401, Obtain User ID; S402. Load the real footprint image from the user ID history cache; S403. Add the historical cached real footprint image to the second footprint image data; S404. Determine the second prompt vector based on the prompt word template and the second footprint image data; S405. Input the second prompt vector into the footprint evaluation reasoning model to obtain the plantar evaluation result.

[0039] Step S404 is similar to step S203. A second prompt vector is generated through the prompt word template, and the second prompt vector is input into the footprint evaluation reasoning model to obtain the sole evaluation result.

[0040] Specifically, the foot assessment results include foot length, foot width, arch type, and recommended shoe type text. The second cue vector serves as the input to the footprint assessment inference model, guiding the model to predict the foot assessment results based on the second footprint image data.

[0041] In the fine-tuning phase of the footprint evaluation inference model, the first footprint image data is input into the pre-trained image language model for training, enabling the footprint evaluation inference model to perform foot type classification tasks. In the deployment phase of the footprint evaluation inference model, the second footprint image data is input into the footprint evaluation inference model to predict the foot assessment results. This embodiment of the application fine-tunes the pre-trained image language model to obtain a footprint evaluation inference model capable of performing foot type classification tasks. After the user takes a picture of a foot model to obtain a real footprint image, they input it into the footprint evaluation inference model to obtain the foot assessment results, which output the user's foot length, foot width, arch type, and shoe selection suggestions.

[0042] Each time a user uploads a real footprint image, the real footprint image is stored in a folder under that user ID, and the footprint evaluation inference model is further fine-tuned based on the feedback text sent by the user.

[0043] After step S400, step S500 is also included: fine-tuning the footprint evaluation inference model based on the feedback text sent by the user; Step S500 includes: S501. Obtain the current feedback text of the user ID; S502. Fine-tune the footprint evaluation reasoning model based on the current feedback text.

[0044] Step S502 specifically includes: determining a fine-tuning strategy based on the current feedback text; if the current feedback text is a negative feedback type, determining a reverse fine-tuning mechanism based on the newly added batch of real footprint images cached by the user ID; if the newly added batch of real footprint images is less than or equal to a preset batch threshold, using an attention weight redistribution mechanism to optimize the model's inference ability; if the newly added batch of real footprint images is greater than the preset batch threshold, using the LoRA mechanism to fine-tune the plantar evaluation inference model. If the current feedback text is a positive feedback type, then the weight parameters of the footprint evaluation inference model remain unchanged.

[0045] Specifically, if the feedback text sent by the user indicates that the current foot evaluation result is inaccurate or the shoes are not suitable, then the current feedback text is determined to be negative feedback; if the feedback text sent by the user indicates that the current foot evaluation result is accurate or the shoes are suitable, then the current feedback text is determined to be positive feedback.

[0046] In this embodiment, the foot assessment inference model supports lifelong learning and self-optimization, continuously receiving new user data and fine-tuning it online. It uses a LoRA mechanism for incremental model updates to avoid the forgetting effect and gradually improve the model's assessment capabilities. Especially during children's growth, foot shape changes frequently; the system can dynamically update the assessment model and track the user's foot development over the long term, providing long-term health guidance.

[0047] Example 2 A foot assessment system based on a large model, such as Figure 2 As shown, it includes a foot imprint unit, a footprint image acquisition unit, and a cloud inference unit.

[0048] The foot impression unit is used to take impressions of a standard foot mold and the user's foot. Specifically, the impression template is made of a deformable material. When acquiring foot data, the user places both feet on the impression template and presses it until a footprint is formed on the template.

[0049] The footprint image acquisition unit is used to acquire first footprint image data and second footprint image data. The footprint image acquisition unit uses a high-resolution camera module and is combined with an edge lighting device to ensure that the footprint outline and arch features are clearly visible.

[0050] The cloud-based inference unit is used to construct a training set using the first footprint image data to fine-tune the open-source image language model, thereby obtaining a footprint evaluation inference model, and to input the second footprint image data into the footprint evaluation inference model to obtain a plantar evaluation result.

[0051] The solution of this application has been described in detail above with reference to the accompanying drawings. In the above embodiments, the descriptions of each embodiment have different emphases; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art should also understand that the actions and modules involved in the specification are not necessarily essential to this application. Furthermore, it is understood that the steps in the method of this application embodiment can be adjusted, combined, and deleted according to actual needs, and the modules in the device of this application embodiment can be combined, divided, and deleted according to actual needs.

[0052] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0053] Alternatively, this application may be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing executable code (or computer program, or computer instruction code) thereon, which, when executed by a processor of an electronic device (or electronic device, server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0054] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the present application can be implemented as electronic hardware, computer software, or a combination of both.

[0055] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0056] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A foot plantar assessment method based on a large model, characterized in that, include: Collect the first footprint image data; The first footprint image data includes N footprint images; The footprint image is a footprint image obtained by taking a standard footprint from a template, where N is an integer greater than or equal to 1; The first footprint image data was used to construct a training set to fine-tune the open-source image language model, resulting in a footprint evaluation inference model. Acquire second footprint image data; the second footprint image data includes a real footprint image; the real footprint image is a footprint image obtained by taking a mold of the user's foot on a template. The second footprint image data is input into the footprint evaluation inference model to obtain the sole evaluation result; the sole evaluation result includes foot length, foot width, arch type and shoe type recommendation text; The footprint evaluation reasoning model is fine-tuned based on the feedback text sent by the user.

2. The plantar evaluation method based on a large model according to claim 1, characterized in that, The acquisition of the first footprint image data specifically includes: Place the standard foot mold on the template to take the mold; Take a picture of the footprint on the template to obtain the footprint image of the foot model; The foot mold and footprint images are preprocessed; The footprint image is annotated to obtain the first footprint image data.

3. The plantar evaluation method based on a large model according to claim 2, characterized in that, The preprocessing of the footprint image includes: An affine transformation is performed on the foot mold footprint image with the direction perpendicular to the plate surface of the template as the frontal viewing direction; The footprint image is subjected to grayscale processing and region segmentation to obtain the footprint image that retains the footprint area.

4. The plantar evaluation method based on a large model according to claim 1, characterized in that, The open-source image language model was fine-tuned using a training set constructed from the first footprint image data to obtain a footprint evaluation inference model, which specifically includes: The first footprint image data is divided into a training set and a validation set; Set LoRA; A first prompt vector is determined based on the prompt word template and the first footprint image data; the prompt word template includes data type prompts, foot length and width prediction prompts, arch type classification prompts, and shoe type recommendation prompts. Load the training set, calculate the loss between the prediction results of the pre-trained large model and the labels, and perform iterations; The footprint evaluation inference model is obtained by exporting the finely tuned pre-trained large model.

5. The plantar evaluation method based on a large model according to claim 4, characterized in that, The first footprint image data is used to construct a footprint image training set, and the foot model footprint image includes foot arch region annotation and foot length annotation.

6. The plantar evaluation method based on a large model according to claim 5, characterized in that, The second footprint image data is input into the footprint evaluation inference model to obtain the plantar assessment result, specifically including: Get user ID; Load the real footprint image from the user ID history cache; Add the historical cached real footprint images to the second footprint image data; A second prompt vector is determined based on the prompt word template and the second footprint image data; The second prompt vector is input into the footprint evaluation inference model to obtain the plantar evaluation result.

7. The plantar evaluation method based on a large model according to claim 6, characterized in that, The footprint evaluation inference model is fine-tuned based on user feedback text, specifically including: Get the current feedback text for the user ID; The reverse fine-tuning mechanism is determined based on the newly added batch of real footprint images cached by the user ID. If the newly added batch of real footprint images is less than or equal to a preset batch threshold, the attention weight redistribution mechanism is used to optimize the model's inference ability. If the newly added batch of real footprint images is greater than the preset batch threshold, the LoRA mechanism is used to fine-tune the plantar evaluation inference model.

8. A plantar assessment system based on a large model, characterized in that, Used to implement the steps in the plantar assessment method according to any one of claims 1 to 7.

Images