Method for measuring real size of object feature in image and foot segmentation method

Abstract

This invention relates to the field of object size measurement in photographs, and more particularly to a method for measuring the real-world size of object features in an image, comprising: S1: reading an image with a reference object of known size, and identifying the reference object and the target object; S2: establishing a reference object coordinate system and an image coordinate system, and geometrically transforming the points in the reference object coordinate system to the points in the image coordinate system; S3: extending the reference object coordinate system beyond the object's boundary; S4: extracting the measurement points of the target object in the image, obtaining the image coordinates of the corresponding measurement points in the image coordinate system, obtaining the physical scale position data of the target object, calculating the distance between different measurement points based on the corresponding coordinates in the reference object coordinate system, and thus obtaining the size of the target object. It is not necessary for the reference object to completely cover the target object; it only needs to occupy a certain area in the image.

Description

Technical Field

[0001] This invention relates to the field of measuring the size of objects in photographs, and more particularly to a method for measuring the actual size of object features in an image and a method for measuring feet. Background Technology

[0002] Chinese patent CN102890805A discloses a method for obtaining two sets of foot feature values ​​by taking photos with a mobile phone and combining them with physical measurements, and then matching the shoe last size using a piecewise linear formula.

[0003] US Patent No. 11779084B2 discloses a method for measuring the size and shape of a foot through image processing, comprising the following steps: acquiring images of a user's foot and a standard-sized object simultaneously; and a calculation step of calculating foot size or shape information based on the images. The images are captured while the user's foot is at least partially in contact with the object.

[0004] European invention patent EP2641539B1 discloses a method for determining the dimensions of at least one foot, which involves taking a single photo of an A4 sheet of paper and a foot with a mobile phone to measure the size of the foot. The claims explicitly require that "the foot must be placed on a reference object".

[0005] The shortcomings of the existing technology are: it requires the feet to be placed on a reference object, and the lower limit of the size of the reference object is relatively high. If the reference object is too small, it is impossible to measure the subject being photographed. Summary of the Invention

[0006] The purpose of this invention is to provide a method for measuring the real size of object features in an image, in order to solve the technical problem in the prior art that the lower limit of the size of the reference object is required.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for measuring the actual size of object features in an image, comprising: S1: Read an image with a reference object of known size and identify the reference object and the target object; S2: Establish a reference coordinate system and an image coordinate system, and make the points in the reference coordinate system correspond to the points in the image coordinate system in a geometric transformation relationship; S3: Extend the reference coordinate system beyond the object boundary of the target object. The reference object surface coordinate system is a U / V coordinate system. Through the mathematical extrapolation property of the geometric transformation, it extends beyond the physical boundary of the reference object. This extrapolation ensures that pixels in the target object region located outside the visual boundary of the reference object can be mapped to the parameter coordinate system through the geometric transformation and obtain valid coordinate values ​​where U coordinate is less than 0, U coordinate is greater than 1, and V coordinate is less than 0 or V coordinate is greater than 1. Each unit U coordinate in the extrapolated region corresponds to the physical dimension of the reference object in the U direction. The extraction of the target object features is processed uniformly in both the extrapolated and non-extrapolated regions. When the target object completely or mostly extends beyond the physical boundary of the reference object, the method still outputs the target object feature values. S4: Extract the measurement points of the target object in the image, obtain the image coordinates of the corresponding measurement points in the image coordinate system, obtain the reference coordinates of the corresponding image coordinates in the reference coordinate system according to the geometric transformation relationship, obtain the physical scale position data of the target object, calculate the distance between different measurement points according to the corresponding coordinates in the reference coordinate system, and then obtain the size of the target object. The target object is the object to be measured in the image, the measurement point is a defined point needed to measure the target object, and the image with a reference object of known size is obtained by placing the reference object next to or above the target object.

[0008] By employing the aforementioned technical solution, utilizing the geometric transformations of the coordinate system and the coordination of measurement points, combined with the known dimensions of the reference object, it is not necessary for the reference object to completely cover the target object; it is sufficient for the reference object to occupy a certain area in the image. Through the geometric transformation and extension of the coordinate system, the size information of the reference object can be transferred to the entire image space where the target object is located, thereby accurately calculating the actual size of the target object. This method effectively lowers the lower limit requirement for the size of the reference object, enabling accurate measurement even if the size of the reference object is smaller than the target object. This solves the technical problems existing in the background technology and improves the flexibility and practicality of measuring the size of objects in images.

[0009] Furthermore, when there are multiple input images, multiple frames are extracted, and the multi-frame features of the multiple frames are fused across frames. The cross-frame fusion method includes scalar median and / or weighted average and / or curve-aligned point-by-point averaging and / or keypoint Kalman smoothing and / or 3D point cloud multiview. Figure 3 Keratinization and / or 3D point cloud ICP alignment followed by fusion and / or attention-based multi-frame feature aggregation.

[0010] By adopting the above technical solution and fusing multiple frames of images, more accurate data can be obtained.

[0011] Furthermore, when the number of input images N equals 1, the cross-frame fusion step adopts an identity mapping, directly using the measurement result of the single frame as the output, without introducing any multi-frame fusion operations.

[0012] With the above technical solution, when the number of input images is 1, without using any multi-frame-dependent computing modules such as timing information, optical flow, and Kalman filtering, it is still possible to output all the foot feature values ​​as described in claim 1.

[0013] Furthermore, the multi-frame images are scored based on one or more of the following: image clarity, lighting, focus state, motion blur degree, complete visibility of reference objects, complete visibility of target objects, and shooting angle. A score threshold is set, and images with scores below the threshold are discarded or have their weight reduced during the fusion process.

[0014] Furthermore, the geometric transformation relationship is a homography matrix H, which is calculated from the image coordinates and physical coordinates of four or more corner points of the reference object.

[0015] By adopting the above scheme, the solution of geometric transformation relationships does not depend on camera intrinsic parameters (focal length, principal point, lens distortion coefficient) and camera extrinsic parameters (camera pose). The method does not require pre-calibration of the camera and can be run directly on any uncalibrated ordinary digital camera because the homography matrix H only requires the image-physical coordinate pairs of the four corner points of the reference object to be solved. This is significantly different from existing 3D vision methods such as SfM, monocular depth, and PnP, which all require pre-calibration of the camera using tools such as checkerboard patterns.

[0016] Furthermore, depth estimation is used to compensate for the height offset of the target object from being on the reference plane. The source of the depth estimation includes at least one of the following: monocular depth neural network, binocular stereo vision, multi-view stereo vision, depth map acquired by a structured light sensor, and depth map acquired by a time-of-flight sensor.

[0017] Furthermore, the cross-frame fusion is a weighted median or weighted average, where the weights are determined by the quality score of each frame.

[0018] Furthermore, multi-view is adopted. Figure 3 Keratinization recovers the shape of a 3D object from feature points at different viewpoints; the multi-view... Figure 3 Angularization minimizes reprojection error.

[0019] Furthermore, the target object is an image of a human foot, and the measurement points are taken from foot position information, which includes at least one of foot mask, foot outline, and foot key points.

[0020] Furthermore, the foot feature values ​​are calculated in the reference coordinate system, which includes a U-axis and a V-axis. The feature values ​​include at least one of the following methods: the projection range in the U-axis direction multiplied by the physical size of the reference object in the U-direction to obtain the foot width; the projection range in the V-axis direction multiplied by the physical size of the reference object in the V-direction to obtain the foot length; the longer side of the smallest area circumscribed rotating rectangle is used as the foot length, and the shorter side is used as the foot width.

[0021] Furthermore, at least one foot feature is extracted from the foot position information, the foot feature including (a) a one-dimensional scalar size of the foot; (b) a foot axis direction vector; (c) coordinates of anatomical key points; (d) a foot boundary feature curve, the boundary feature curve being selected from at least one of the toe curve, instep curve, medial foot curve, lateral foot curve, and heel curve; (e) a cross-sectional width sequence and the inflection point positions on the sequence; (f) 3D shape parameters of the foot; and (g) any one or more of the foot classification labels.

[0022] Furthermore, the foot position information extraction uses a supervised-trained foot-specific segmentation network trained on a foot segmentation dataset, which includes foot images taken from top, side, and oblique angles.

[0023] Furthermore, the foot features include foot length, which is defined as the distance between the two endpoints in the longest direction of the foot. The method for calculating the foot length includes at least one of the following: the distance between extreme points on the principal axis of principal component analysis, the longer side of the circumscribed rotation rectangle with the minimum area, the product of the extreme range of the V-axis and the physical size in the V direction in the reference object UV coordinate system, and the Euclidean distance from the key point of the big toe to the key point of the last end of the heel.

[0024] Furthermore, the foot feature includes foot width, which is defined as including at least one of the following: the length of the line connecting the first and fifth metatarsophalangeal joints, the extreme range of the U-axis in the reference UV coordinate system, the shorter side of the smallest area circumscribed rotating rectangle, and the cross-sectional width at the widest point of the foot.

[0025] Furthermore, the foot features include a toe curve, which is defined as a continuous planar curve of the contours of the five toes. The method for extracting the toe curve includes at least one of the following: finding the boundary contour segment in the forefoot direction on the foot mask, performing spline interpolation through the key points of the five toes, and extracting the boundary along the contour lines near the maximum value of V in the parametric coordinate system.

[0026] Furthermore, the foot features include a cross-sectional width sequence, defined as a sequence of foot widths on multiple cross-sections uniformly sampled along the long axis of the foot, and further include inflection point positions identified from the sequence, the inflection point positions being selected from at least one of the narrowest point of the arch, the widest point of the metatarsophalangeal joint, and the ankle position.

[0027] A method for removing a foot segmentation mask, based on the aforementioned foot size measurement method, includes: T1: Mark the outline of the target object into multiple measurement points, and the multiple measurement points form a point cloud. Read the position of the point cloud in the image coordinate system and the reference object coordinate system, and read the principal axis direction vector and minor axis direction vector of each measurement point in the point cloud. T2: Divide the point cloud into K equal-length bins along the main axis, where K is greater than or equal to 10. Calculate the extreme range along the minor axis for each bin as the cross-sectional width of that bin, and obtain a width array of length K.

[0028] T3: Smooth the width array; T4: Take the position of the global maximum value of the smoothed width array as the forefoot position; T5: Starting from the forefoot position, scan the smoothed width array towards both ends of the main axis, and detect at least one of the following modes to determine the ankle position: the position of the last local minimum in the three-segment local minimum mode; the position where the width first drops below a predetermined proportion of the width of the forefoot position; T6: Divide the ankle position along the main axis and discard the point on one side of the leg at the ankle position; T7: Output the clipped point cloud to obtain the outline of the foot.

[0029] The ankle and toe keypoints output by the human pose estimation model are used to assist or verify the foot segmentation results.

[0030] In summary, the present invention has the following advantages compared with the prior art: 1. It effectively lowers the lower limit requirement for the size of the reference object. The reference object does not need to completely cover the target object. The size information can be transmitted through coordinate system extension and geometric transformation as long as the reference object occupies a certain area of ​​the image. This solves the problem that the reference object is too small to be measured in the existing technology and improves the flexibility of the measurement scenario. 2. By fusing multiple frames of images across frames (such as weighted median, weighted average, etc.) and using a quality scoring mechanism (based on dimensions such as sharpness, lighting, and focus), high-quality images are selected and their weights are dynamically adjusted, combined with multi-view... Figure 3Keratinization restores 3D shape, significantly improving the accuracy and robustness of measurement data and reducing the impact of single-frame image errors; 3. A depth estimation method is introduced to compensate for the height offset of the target (supporting multiple depth sources such as monocular depth neural networks and binocular stereo vision), adapting to the measurement needs in non-planar scenes and expanding the applicability of the method. 4. A dedicated segmentation network (covering multiple shooting angles such as top view, side view, and oblique view) was designed for foot measurement, along with multi-dimensional feature extraction (foot length, foot width, contour curve, cross-sectional width sequence, etc.) and ankle pruning optimization steps, which realizes the accurate acquisition of detailed features and meets the needs of practical application scenarios such as shoe last matching. 5. The geometric transformation employs homography matrix and parametric coordinate system extrapolation, ensuring the mathematical rigor of the coordinate system extension. Even if the foot completely or mostly extends beyond the reference object boundary, valid feature values ​​can still be output. This invention combines practicality and innovation, providing an efficient and accurate solution for measuring object dimensions in images, especially foot measurements, and has broad application prospects. Attached Figure Description

[0031] Figure 1 The image shows the reference object and the foot in the same frame; Figure 2 A schematic diagram of the UV parameter coordinate system and the foot measurement extrapolated to the reference object; Figure 3 A schematic diagram of the characteristic curves of the foot; Figure 4 This is a schematic diagram showing the distribution of the cross-sectional width array along the principal axis; Figure 5 This is a schematic diagram illustrating the construction of the UV parameter coordinate system on the top perspective view; Figure 6 A schematic diagram for calculating the leg length in UV space; Figure 7 A schematic diagram for calculating foot width in UV space; Figure 8 This is a perspective diagram of a side-view shooting scene. Figure 9 This is a schematic diagram illustrating the construction of the UV parameter coordinate system on a side perspective view. Figure 10 A comparison chart of cross-frame fusion methods; Figure 11 Illustration of application scenarios for shoes. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0033] Example 1, see Figure 1-10 A method for measuring the actual size of object features in an image, wherein the object being measured in the image is the target object, and in this embodiment, foot features are selected as the target object for measurement.

[0034] A method for measuring the actual size of an object in an image, comprising: S0: Preparation: Take pictures and input the pictures into the system. When taking pictures, place the target and reference objects to be measured at the shooting position. The reference objects can be placed next to the target objects or at the bottom of the target objects.

[0035] In this embodiment, the target object is placed next to a reference object for shooting and measurement. Multiple angles are selected for shooting, including but not limited to top view, front view, side view, bottom view, isometric view, and rear view. The system can read the video and extract key frame images from the recorded video as the images to be read.

[0036] The reference material can be standardized paper, including at least one of the ISO 216 series, the North American ANSI series, and the Japanese JIS series.

[0037] The reference object can be a standard-sized rigid object or a specially designed marking pattern; the rigid object includes at least one of the following: cards (such as credit cards, ID cards), circulating coins, stamps, and business cards conforming to ISO / IEC 7810 ID-1 specifications; the marking pattern includes at least one of the following: AprilTag, ArUco mark, ChArUco checkerboard, QR code, and custom color marking pattern.

[0038] The reference point can also be another known part of the user's body, whose physical dimensions have been measured in advance through other means.

[0039] In this embodiment, the reference objects are cards with specific specifications, such as credit cards and ID cards.

[0040] Quality assessment: Each frame of the image is scored based on factors such as image sharpness, lighting, focus, motion blur, visibility of reference objects, visibility of the target object (in this embodiment, the target object is the foot), and shooting angle. A score threshold is set, and images below the set threshold are discarded or have their weight reduced.

[0041] S1: Read the image with a reference object of known size, and identify the reference object and the target object (in this embodiment, the target object is the foot part of the human body); When there are multiple input images, multiple frames are extracted, and the multi-frame features of the multiple frames are fused across frames. The cross-frame fusion method includes scalar value median and / or weighted average and / or curve-aligned point-by-point averaging and / or keypoint Kalman smoothing and / or 3D point cloud multiview. Figure 3 Angularization and / or 3D point cloud ICP alignment followed by fusion and / or attention-based multi-frame feature aggregation; S2: Establish a reference coordinate system and an image coordinate system, and make the points in the reference coordinate system correspond to the points in the image coordinate system in a geometric transformation relationship; In this embodiment, the geometric transformation relationship is the homography matrix H, which is calculated from the image coordinates and physical coordinates of four or more corner points of the reference object.

[0042] S3: Extend the reference coordinate system beyond the object's boundary; The reference object surface parameter coordinate system is extended beyond the physical boundary of the reference object through the mathematical extrapolation property of the geometric transformation. The extrapolation ensures that pixels in the foot region located outside the visual boundary of the reference object can be mapped to the parameter coordinate system through the geometric transformation and obtain valid coordinate values ​​with U coordinate less than 0, U coordinate greater than 1, V coordinate less than 0, or V coordinate greater than 1. Each unit U coordinate in the extrapolated region corresponds to the physical dimension of the reference object in the U direction. The extraction of foot features is processed uniformly in the extrapolated and non-extrapolated regions. When the foot extends completely or mostly beyond the physical boundary of the reference object, the method still outputs foot feature values.

[0043] S4: Extract the measurement points of the target object in the image, obtain the image coordinates of the corresponding measurement points in the image coordinate system, obtain the reference coordinates of the corresponding image coordinates in the reference coordinate system according to the geometric transformation relationship, obtain the physical scale position data of the target object, calculate the distance between different measurement points according to the corresponding coordinates in the reference coordinate system, and then obtain the size of the target object.

[0044] The measurement points are taken from foot position information, including at least one of foot mask, foot outline, and foot key points.

[0045] The extraction of foot position information uses a supervised-trained foot-specific segmentation network trained on a foot segmentation dataset that includes foot images taken from top, side, and oblique angles.

[0046] The foot feature value is calculated in the reference coordinate system, which includes a U-axis and a V-axis. The feature value includes at least one of the following methods: the projection range in the U-axis direction multiplied by the physical size of the reference object in the U-direction to obtain the foot width; the projection range in the V-axis direction multiplied by the physical size of the reference object in the V-direction to obtain the foot length; the longer side of the smallest area circumscribed rotating rectangle is used as the foot length, and the shorter side is used as the foot width.

[0047] Reference Figure 6 , Figure 7 The formula for calculating foot length is: ; The formula for calculating foot width in UV space is as follows: .

[0048] The foot features include (a) a one-dimensional scalar dimension of the foot; (b) a vector of the foot's axial direction; (c) coordinates of anatomical key points; (d) a foot boundary feature curve, which is selected from at least one of the following: toe curve, instep curve, medial foot curve, lateral foot curve, and heel curve; (e) a cross-sectional width sequence and the inflection point positions on the sequence; (f) 3D shape parameters of the foot; and (g) foot classification labels.

[0049] In this embodiment, the foot features also include foot length, foot width, toe curve, and cross-sectional width sequence. The foot length is defined as the distance between the two endpoints in the longest direction of the foot. The calculation method for the foot length includes at least one of the following: the distance between extreme points on the principal axis of principal component analysis, the longer side of the minimum area circumscribed rotating rectangle, the product of the extreme range of the V-axis and the physical size in the V direction in the UV coordinate system of the reference object, and the Euclidean distance from the key point of the big toe to the key point of the last end of the heel. The definition of foot width includes at least one of the following: the length of the line connecting the first and fifth metatarsophalangeal joints, the extreme range of the U-axis in the UV coordinate system of the reference object, the shorter side of the smallest area circumscribed rectangle of revolution, and the cross-sectional width at the widest point of the foot; The toe curve is defined as a continuous planar curve of the contours of the five toes. The method for extracting the toe curve includes at least one of the following: finding the boundary contour segment in the forefoot direction on the foot mask, spline interpolation through the key points of the five toes, and extracting the boundary along the contour lines near the maximum value of V in the parametric coordinate system. The cross-sectional width sequence is defined as a sequence of foot widths on multiple cross-sections uniformly sampled along the long axis of the foot, and further includes inflection point positions identified from the sequence, the inflection point positions being selected from the narrowest point of the arch, the widest point of the metatarsophalangeal joint, and the ankle position.

[0050] When the captured photo is a single still image, cross-frame feature fusion is not required, and this embodiment uses identity mapping. The sources of a single still image include at least one of the following: photos taken by the user, representative frames extracted from videos, images uploaded or pushed by third parties, the current frame captured by the camera in real time, and historical images stored on the client or server.

[0051] A method for removing a foot segmentation mask, based on the aforementioned foot size measurement method, includes: T1: Mark the outline of the target object into multiple measurement points, and the multiple measurement points form a point cloud. Read the position of the point cloud in the image coordinate system and the reference object coordinate system, and read the principal axis direction vector and minor axis direction vector of each measurement point in the point cloud. The principal axis and minor axis refer to the major axis and minor axis directions of the foot itself obtained by principal component analysis (PCA) of the foot point cloud. T2: Divide the point cloud into K equal-length bins along the main axis, where K is greater than or equal to 10. Calculate the extreme range along the minor axis for each bin as the cross-sectional width of that bin, and obtain a width array of length K.

[0052] T3: Smooth the width array; T4: Take the position of the global maximum value of the smoothed width array as the forefoot position; T5: Starting from the forefoot position, scan the smoothed width array towards both ends of the main axis, and detect at least one of the following modes to determine the ankle position: the position of the last local minimum in the three-segment local minimum mode; the position where the width first drops below a predetermined proportion of the width of the forefoot position; T6: Divide the ankle position along the main axis and discard the point on one side of the leg at the ankle position; T7: Output the clipped point cloud to obtain the outline of the foot.

[0053] Furthermore, the ankle and toe keypoints output by the human pose estimation model are used to assist or verify the foot segmentation results.

[0054] Reference Figure 11 A method for customizing shoes, comprising the following steps: A1: Photo upload. The user takes a photo of their feet with their mobile phone and uploads a reference object of known size. In this embodiment, a credit card is used as the reference object. A2: Foot measurement, obtain the foot feature information after measurement according to the foot measurement method described above; A3: Shoe last generation, and the generation of 3D shoe last images based on foot feature information and user requirements; A4: Based on the shoe last and the user's needs, generate shoe styles and provide an AR try-on module for users to experience and select. After the user places an order, the merchant receives the user's requirements and begins manufacturing on demand.

[0055] In the specific implementation process, the user first takes a picture of a standard-sized reference object (such as a credit card) along with their foot to obtain an image containing both. After the system performs a quality assessment of the image (clarity, lighting, integrity, etc.), it enters the core processing stage.

[0056] S1-S3: Establishing and extending the reference coordinate system The system identifies reference objects in the image (such as the four corners of a credit card), calculates the homography matrix H based on their known physical dimensions and image coordinates, and thus establishes the geometric transformation relationship between the image coordinate system and the UV parameter coordinate system of the reference object.

[0057] The key step lies in mathematically extrapolating this coordinate system to cover the entire area where the foot is located. This means that even if part or all of the foot is outside the physical boundary of the reference card, every pixel on the foot mask can be mapped to the reference coordinate system through matrix H and obtain its (U, V) coordinate value. Furthermore, since the physical dimensions of the reference object in the U and V directions (such as the length and width of a credit card) are known, each unit U or V coordinate in the extrapolated area of ​​the coordinate system also corresponds to the actual physical length. This lays the foundation for subsequently calculating the physical dimensions directly from the coordinates.

[0058] S4: Extract foot features in the reference coordinate system All calculations of foot features are performed within the established reference coordinate system, thus directly obtaining the physical dimensions in the real world. Specific methods include: One approach is to calculate the extreme coordinates (U_max and U_min) of all pixels in the foot along the U-axis, then the foot width = (U_max - U_min) × (physical dimension of the reference object in the U direction).

[0059] Similarly, the extreme range of the V-axis coordinates can be calculated, and multiplied by the physical dimension in the V direction to obtain the foot length. Another method is to calculate the minimum area circumscribed by the rotation rectangle based on the foot point cloud, and its long and short sides can be directly used as estimates of the foot length and width.

[0060] In the parametric coordinate system, contour lines of the V coordinate that are close to the maximum value (i.e. the toe) can be extracted. This continuous curve formed by high V coordinate points is the toe curve.

[0061] The foot is divided into multiple equidistant intervals (bins) along the long axis of the foot (such as the U-axis or the principal axis determined by principal component analysis). Within each interval, the range of coordinates of all points along the short axis (such as the V-axis) is calculated, and then multiplied by the corresponding physical size scaling factor to obtain the physical width of the cross-section. This forms a sequence of width variations along the length of the foot, which is used to locate inflection points such as the arch and forefoot.

[0062] After obtaining accurate foot contour and feature data, the system executes the foot segmentation masking method (T1-T7), which automatically segments the foot and leg using the morphological features of the width array (such as the widest part of the forefoot and the ankle contraction point), and can use key points of the human pose estimation model for auxiliary verification.

[0063] Finally, the extracted foot feature data (physical dimensions, contour curves, inflection point positions, etc.) are input into the shoe last generation algorithm, combined with the shoe style selected by the user, to generate a personalized 3D shoe last model. Users can preview the wearing effect through the AR module, and place an order after confirmation. The data goes directly to the production end for customized manufacturing.

[0064] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0065] It should be understood that although the present invention may use terms such as first, second, third, etc., to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of the present invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for measuring the real-world dimensions of object features in an image, characterized in that, include: S1: Read an image with a reference object of known size and identify the reference object and the target object; S2: Establish a reference coordinate system and an image coordinate system, and make the points in the reference coordinate system correspond to the points in the image coordinate system in a geometric transformation relationship; S3: Extend the reference coordinate system beyond the object boundary of the target object. The reference object surface coordinate system is a U / V coordinate system, which is extended beyond the physical boundary of the reference object through the mathematical extrapolation property of the geometric transformation. The extrapolation ensures that pixels in the target object region located outside the visual boundary of the reference object can be mapped to the parameter coordinate system through the geometric transformation and obtain valid coordinate values ​​where U coordinate is less than 0, U coordinate is greater than 1, and V coordinate is less than 0 or V coordinate is greater than 1. Each unit U coordinate in the extrapolated region corresponds to the physical dimension of the reference object in the U direction. The extraction of the target object features is processed uniformly in the extrapolated region and the non-extrapolated region. When the target object completely or mostly extends beyond the physical boundary of the reference object, the method still outputs the target object feature values. S4: Extract the measurement points of the target object in the image, obtain the image coordinates of the corresponding measurement points in the image coordinate system, obtain the reference coordinates of the corresponding image coordinates in the reference coordinate system according to the geometric transformation relationship, obtain the physical scale position data of the target object, calculate the distance between different measurement points according to the corresponding coordinates in the reference coordinate system, and then obtain the size of the target object. The target object is the object to be measured in the image, the measurement point is a defined point needed to measure the target object, and the image with a reference object of known size is obtained by placing the reference object next to or above the target object.

2. The method for measuring the actual size of object features in an image according to claim 1, characterized in that, When there are multiple input images, multiple frames of images are extracted, and the multi-frame features of the multiple frames of images are fused across frames. The cross-frame fusion method includes scalar value median and / or weighted average and / or curve alignment followed by point-by-point averaging and / or key point Kalman smoothing and / or 3D point cloud multi-view triangulation and / or 3D point cloud ICP alignment followed by fusion and / or attention-based multi-frame feature aggregation.

3. The method for measuring the actual size of object features in an image according to claim 1, characterized in that, When the number of input images N equals 1, the cross-frame fusion step uses an identity mapping, directly using the measurement result of the single frame as the output, without introducing any multi-frame fusion operations.

4. The method for measuring the actual size of object features in an image according to claim 2, characterized in that, The multi-frame images are scored based on one or more of the following factors: image clarity, lighting, focus state, motion blur degree, complete visibility of reference objects, complete visibility of target objects, and shooting angle. A score threshold is set, and images with scores below the threshold are discarded or have their weight reduced during the fusion process.

5. The method for measuring the actual size of object features in an image according to claim 1, characterized in that, The geometric transformation relationship is a homography matrix H, which is calculated from the image coordinates and physical coordinates of four or more corner points of the reference object.

6. The method for measuring the actual size of object features in an image according to claim 1, characterized in that, The depth estimation is further used to compensate for the height offset of the target object when it is not on the reference plane. The source of the depth estimation includes at least one of the following: monocular depth neural network, binocular stereo vision, multi-view stereo vision, depth map obtained by structured light sensor, and depth map obtained by time-of-flight sensor.

7. The method for measuring the actual size of object features in an image according to claim 2, characterized in that, The cross-frame fusion is a weighted median or weighted average, where the weights are determined by the quality score of each frame.

8. The method for measuring the actual size of object features in an image according to any one of claims 1-7, characterized in that, The target object is the human foot, the target object feature is the foot feature, the measurement point is taken from the foot position information, and the foot position information includes at least one of the following: foot mask, foot outline, and foot key points.

9. A method for measuring the actual size of object features in an image according to claim 8, characterized in that, The foot feature value is calculated in the reference coordinate system, which includes a U-axis and a V-axis. The feature value includes at least one of the following methods: the projection range in the U-axis direction multiplied by the physical size of the reference object in the U-direction to obtain the foot width; the projection range in the V-axis direction multiplied by the physical size of the reference object in the V-direction to obtain the foot length; the longer side of the smallest area circumscribed rotating rectangle is used as the foot length, and the shorter side is used as the foot width.

10. A method for measuring the actual size of object features in an image according to claim 8, characterized in that, The foot features include (a) a one-dimensional scalar dimension of the foot; (b) a vector of the foot's axial direction; (c) coordinates of anatomical key points; (d) a foot boundary feature curve, which is selected from at least one of the following: toe curve, instep curve, medial foot curve, lateral foot curve, and heel curve; (e) a cross-sectional width sequence and the inflection point positions on the sequence; (f) 3D shape parameters of the foot; and (g) any one or more of the following foot classification labels.

11. A method for measuring the actual size of object features in an image according to claim 8, characterized in that, The foot position information extraction uses a supervised-trained foot-specific segmentation network trained on a foot segmentation dataset, which includes foot images taken from top, side, and oblique angles.

12. The method for measuring the actual size of object features in an image according to claim 8, characterized in that, The foot features include foot length and / or foot width and / or toe curve and / or cross-sectional width sequence.

13. A method for removing a foot segmentation mask based on the measurement method of any one of claims 1-7, characterized in that, include: T1: Mark the outline of the target object into multiple measurement points, and the multiple measurement points form a point cloud. Read the position of the point cloud in the image coordinate system and the reference object coordinate system, and read the principal axis direction vector and minor axis direction vector of each measurement point in the point cloud. T2: Divide the point cloud into K equal-length bins along the main axis, where K is greater than or equal to 10. Calculate the extreme range along the minor axis for each bin as the cross-sectional width of that bin, and obtain a width array of length K. T3: Smooth the width array; T4: Take the position of the global maximum value of the smoothed width array as the forefoot position; T5: Starting from the forefoot position, scan the smoothed width array towards both ends of the main axis, and detect at least one of the following modes to determine the ankle position: the position of the last local minimum in the three-segment local minimum mode; the position where the width first drops below a predetermined proportion of the width of the forefoot position; T6: Divide the ankle along the main axis and discard the points on one side of the leg at the ankle position; T7: Output the clipped point cloud to obtain the outline of the foot; The ankle and toe keypoints output by the human pose estimation model are used to assist or verify the foot segmentation results.

Images