Intelligent photographing system based on crowd classification

Abstract

The application provides an intelligent photo system based on crowd classification, relates to the technical field of image processing, receives a crowd category instruction input by a user, acquires a photo template set associated with the crowd category instruction in response to the crowd category instruction, the photo template set comprises thumbnails composed of a plurality of photo templates, obtains a target template selected by the user based on a plurality of thumbnails in the photo template set, collects an original image of the user, performs pose analysis based on the original image and the target template, generates an optimal mapping relationship, and generates an intermediate image according to the optimal mapping relationship. Not only the realness and visual effect of the synthesized image are greatly improved, but also the adaptability and robustness of the system are enhanced, so that the system can stably output high-quality photo images under different scenes and requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an intelligent photograph system based on crowd classification. BACKGROUND

[0002] In the existing photographing and AI image processing technology, with the development of deep learning and generative AI technology, some intelligent photograph systems have begun to introduce AI models to artistically render the original images captured by users to generate more visually expressive photograph works. For example, some existing systems attempt to build an AI workstation end, integrate visual AI workflow tools such as ComfyUI, and combine large model base training, LoRA fine-tuning, face fusion, etc. to achieve stylized conversion and fine rendering of images. Such systems usually adopt a cloud collaborative architecture, upload the original images captured by users through the client to the server, and then call the pre-trained model on the AI workstation end to complete image generation, and return the results to the client for the user to download.

[0003] Users usually generate photographs by selecting preset templates, and the system directly performs image synthesis or morphing processing on the original image and the target template to obtain an intermediate image. By building an AI workstation end, integrating visual AI workflow tools such as ComfyUI, and combining large model base training, LoRA fine-tuning, face fusion, etc., stylized conversion and fine rendering of images are achieved to obtain a rendered image. Such systems usually adopt a cloud collaborative architecture, upload the original images captured by users through the client to the server, and then call the pre-trained model on the AI workstation end to complete image generation, and return the results to the client for the user to download.

[0004] The existing technology usually only performs simple coordinate mapping based on key point detection when generating an intermediate image, and completes image deformation through a traditional interpolation algorithm (such as thin-plate spline TPS). This process does not introduce structural perception analysis of the original image, cannot distinguish important edges from texture noise, and causes distortion of key contours during non-rigid transformation. At the same time, due to the lack of differentiated processing mechanisms for semantic regions, the system treats all regions equally during deformation, cannot dynamically adjust the weight according to the visual importance of the region, and thus affects the overall quality of the generated image. SUMMARY

[0005] The present application aims to at least partially solve one of the technical problems in the related art.

[0006] To this end, the present application proposes an intelligent photograph system based on crowd classification to solve the above technical problems.

[0007] To achieve the above purpose, the embodiments of the present application propose an intelligent photograph system based on crowd classification, comprising:

[0008] An instruction receiving module is configured to receive a user-input crowd category instruction.

[0009] A photo template obtaining module is configured to obtain, in response to the crowd category instruction, a photo template set associated with the crowd category instruction, the photo template set including thumbnails composed of a plurality of photo templates.

[0010] A target template obtaining module is configured to obtain a target template selected by a user based on the plurality of thumbnails in the photo template set.

[0011] A collecting module is configured to collect an original image of the user.

[0012] An image analyzing module is configured to perform pose analysis based on the original image and the target template, generate an optimal mapping relationship, and generate an intermediate image according to the optimal mapping relationship.

[0013] A synthesizing module is configured to perform layer superimposition processing on the intermediate image to generate a synthesized image.

[0014] Further, each photo template includes a person pose key point and a person pose key point coordinate.

[0015] Further, the image analyzing module includes the following sub-modules.

[0016] A pose analyzing sub-module is configured to perform human body pose analysis on the original image to obtain a person joint point and a person joint point coordinate, and construct an original pose description graph according to the person joint point and the person joint point coordinate, the original pose description graph including a plurality of first nodes and first edges, wherein each person joint point serves as a first node of the original pose description graph, each first node marks a person joint point type and a first included angle, and a skeletal connection between adjacent person joint points serves as a first edge, and each first edge marks a first connection length between adjacent first nodes and a first coordinate point.

[0017] A converting sub-module is configured to generate a standard pose description graph based on a person pose key point coordinate of the target template, the standard pose description graph including a plurality of second nodes and second edges, wherein each person pose key point serves as a second node of the standard pose description graph, each second node marks a person joint point type and a second included angle, a skeletal connection between adjacent person pose key points serves as a second edge, and each second edge marks a second connection length between adjacent second nodes and a second coordinate point.

[0018] An aligning sub-module is configured to perform alignment inference based on a graph structure of the original pose description graph and the standard pose description graph to obtain an optimal mapping relationship between the first nodes and the second nodes.

[0019] an image processing submodule, configured to generate a sparse point set according to the optimal mapping relationship; perform gradient analysis based on the original image to generate a dynamic weight, perform interpolation weighting on the sparse point set based on the dynamic weight to obtain a target point set, and map the target point set of the original image to the standard pose description graph to obtain an intermediate image.

[0020] Further, the alignment submodule comprises the following units:

[0021] a neighborhood construction unit, configured to extract all adjacent nodes and adjacent edges directly connected based on the first node, and construct a first-order neighborhood structure based on the adjacent nodes and the adjacent edges;

[0022] a neighborhood feature construction unit, configured to construct a neighborhood feature based on the first node, the adjacent nodes and the adjacent edges;

[0023] a matching unit, configured to generate a similarity based on each neighborhood feature in the original pose description graph and the standard pose description graph, take the similarity as a probability, and construct a matching matrix according to the probability;

[0024] an updating unit, configured to perform multiple rounds of updating on the matching matrix to generate a matching update matrix, perform convergence evaluation based on the matching update matrix after each round of updating, and determine an optimal mapping relationship based on the final matching update matrix when a maximum number of updating rounds is reached.

[0025] Further, the convergence evaluation performed after each round of updating comprises:

[0026] determining a Frobenius norm difference value based on the current matching matrix and the matching matrix of the last round; if the Frobenius norm difference value is greater than or equal to a first target value, performing updating of the next round based on the current matching matrix;

[0027] if the Frobenius norm difference value is less than the first target value, determining the second node with the highest probability of the first node in the matching matrix and marking it as a second sub-node, and determining the first node with the highest probability of the second node in the matching matrix and marking it as a first sub-node;

[0028] determining a corresponding adjacent edge based on the first sub-node, determining a first connection length according to the first sub-node and the corresponding adjacent edge, and determining a first included angle based on the first sub-node; determining a corresponding adjacent edge based on the second sub-node, determining a second connection length according to the second sub-node and the corresponding adjacent edge, and determining a second included angle based on the second sub-node;

[0029] determine an angle deviation based on a difference between the first angle and the second angle;

[0030] determine, according to the adjacent edge corresponding to the first sub-node, the adjacent node directly connected to the first sub-node, generate a first connection ratio based on a first connection edge between the first sub-node and the adjacent node, determine, according to the adjacent edge corresponding to the second sub-node, the adjacent node directly connected to the second sub-node, generate a second connection ratio based on a second connection edge between the second sub-node and the adjacent node, and generate a connection ratio difference based on the first connection ratio and the second connection ratio;

[0031] if the first sub-node and the second sub-node have the same character node type, and the angle deviation is less than a second target value and the connection ratio difference is less than a third target value, determine that the first sub-node passes the structural consistency test, otherwise, perform the next round of update based on the current matching matrix;

[0032] statistically determine a proportion of the first sub-nodes that pass the structural consistency test in the original pose description graph, and if the proportion is less than 80%, perform the next round of update based on the current matching matrix;

[0033] determine a candidate matching pair based on the first sub-node and the second sub-node, determine a limb connection ratio change rate based on the candidate matching pair, if the limb connection ratio change rate meets a target condition, determine that the current matching update matrix is in a convergent state, and take the current matching update matrix as a final matching update matrix, otherwise, perform the next round of update based on the current matching matrix.

[0034] Further, in the final matching update matrix, an optimized matching pair is determined according to the probability of each candidate matching pair, and a Hungarian algorithm is performed on the optimized matching pair to obtain an optimal mapping relationship, i.e., a one-to-one correspondence between the first sub-node and the second sub-node.

[0035] Further, a sparse point set is generated according to the optimal mapping relationship, and a dynamic weight is generated based on gradient analysis of the original image, including:

[0036] a semantic region is generated based on the original image;

[0037] a multi-scale gradient intensity field is extracted based on the original image, and a plurality of gradient maps are obtained based on the multi-scale gradient intensity field;

[0038] a structure importance map is obtained based on structure analysis of the plurality of gradient maps, and a structure importance score of each pixel point is obtained based on the structure importance map;

[0039] a semantic modulation factor is determined according to the structure importance score of each semantic region;

[0040] According to a relationship between the semantic modulation factor and / or the structure importance score and a preset threshold condition, a dynamic weight map is generated, and a dynamic weight of a semantic region is determined based on the dynamic weight map.

[0041] Further, the semantic region includes a skin region, a clothing region, and a background region.

[0042] Compared with the prior art, the intelligent photo system based on crowd classification provided in the application constructs a matching matrix and obtains a final matching matrix based on multi-layer convergence evaluation, and generates an optimal node mapping relationship between an original pose description graph and a standard pose description graph according to the final matching matrix. In the convergence evaluation at each level, not only the difference between the current matching matrix and the last round is considered, but also the angle deviation and connection proportion difference between nodes are analyzed in detail, so that even slight structural changes can be accurately captured and processed. This method can effectively avoid the deformation distortion problem caused by simple coordinate mapping when dealing with complex pose transformation or non-rigid deformation. In addition, the optimized matching pairs determined based on the final matching matrix are further refined by using the Hungarian algorithm to establish a one-to-one correspondence between each node, thereby providing high-quality basic data for subsequent image synthesis. This not only greatly improves the realism and visual effect of the synthesized image, but also enhances the adaptability and robustness of the system, so that it can stably output high-quality photo images under different scenes and requirements.

[0043] In the process of mapping the sparse point set to the target point set, the application deeply considers the semantic regions of the original image and generates dynamic weights based on these semantic information, thereby guiding the interpolation algorithm to perform more accurately to generate high-quality intermediate images. The core of this method is to identify and utilize the unique attributes of different semantic regions in the image, and to develop exclusive weight allocation strategies for each region. In this way, not only can the position accuracy of key feature points in the conversion process be significantly improved, but also the natural transition of non-key regions can be ensured, avoiding the blur or distortion problems that may occur in traditional interpolation methods.

[0044] Additional aspects and advantages of the application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0045] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description, including the appended drawings, wherein:

[0046] Figure 1 A structural block diagram of an intelligent photo system based on crowd classification provided by an embodiment of the application. DETAILED DESCRIPTION

[0047] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0048] One embodiment of the present application is described below with reference to the accompanying drawings, which is an intelligent photo system based on crowd classification.

[0049] Figure 1 The structure block diagram of an intelligent photo system based on crowd classification provided by an embodiment of the present application.

[0050] As shown in the drawing, Figure 1 The intelligent photo system based on crowd classification comprises the following modules:

[0051] The instruction receiving module is configured to receive a crowd category instruction input by a user.

[0052] The instruction receiving module is deployed on a client developed based on Unity and runs on an interactive terminal device such as a 32-inch intelligent all-in-one machine, and presents selectable crowd classification options to the user through a graphical user interface. Specifically, the crowd categories include but are not limited to middle-aged men, middle-aged women, small boys, and small girls, each of which corresponds to a photo style with different body shape characteristics, posture preferences, and aesthetic expressions.

[0053] In actual use, the user selects the target crowd category by clicking on the touch screen. For example, when the user is a middle-aged woman, he / she can enter the subsequent process by clicking on the middle-aged woman block. The instruction receiving module detects user interaction events in real time, and once a valid click operation is detected, a corresponding crowd category instruction is generated and encapsulated as a structured data packet containing information such as the category label selected by the user, the timestamp, and the device identifier. The data packet is sent to the cloud server through HTTP or WebSocket protocol for persistent recording, and simultaneously triggers the local process of the client to jump to the photo template acquisition module.

[0054] Further, the instruction receiving module also supports a dynamic expansion mechanism, which can add new crowd categories (such as “young men” and “old women”) through remote server configuration, without the need to update the client application to realize function iteration. At the same time, the module has built-in anti-mis-touch and debounce logic to ensure the accuracy and stability of instruction input in public use scenarios.

[0055] The photo template acquisition module is configured to acquire a photo template set associated with the crowd category instruction in response to the crowd category instruction, wherein the photo template set comprises a plurality of thumbnail photos.

[0056] When the user finishes the crowd category selection, the client sends the category instruction to the server. The server retrieves the photo template set associated with the crowd category from the database according to the preset template mapping relationship table. Each photo template is pre-configured with standard pose description information. Each photo template includes: human pose key points (such as 17 key points such as nose tip, left shoulder, right elbow, and left knee) and human pose key point coordinates. At the same time, each template is also attached with a thumbnail, with a resolution of 160x120 pixels, used to display in the form of grid or carousel in the client interface.

[0057] The photo template set is returned by the server in JSON array format, including template ID, thumbnail URL, template name, applicable scene (such as "business photo", "parent-child photo", "holiday theme"), etc. After receiving the data, the client downloads the thumbnail resources one by one through the asynchronous loading mechanism, and arranges and displays them in the form of visual cards in the photo template selection interface for the user to browse and select.

[0058] The target template acquisition module obtains the target template selected by the user based on the plurality of thumbnails in the photo template set.

[0059] When the photo template set is loaded, all thumbnails are displayed in a grid layout on the screen, and the user can slide through and click to select the desired template through touch operation.

[0060] The acquisition module is used to acquire the original image of the user.

[0061] The high-definition image of 1080x1920 resolution is captured by the camera of the client, and automatic preliminary processing is performed, including image rotation correction (to ensure the vertical screen direction), white balance adjustment, and JPEG compression encoding.

[0062] The image analysis module is used to perform pose analysis based on the original image and the target template, generate an optimal mapping relationship, and generate an intermediate image according to the optimal mapping relationship.

[0063] The image analysis module includes the following sub-modules:

[0064] The pose analysis submodule is used to analyze the human pose of the original image to obtain the human joint points and human joint point coordinates; and construct an original pose description graph according to the human joint points and the human joint point coordinates. The original pose description graph includes a plurality of first nodes and first edges, wherein each human joint point serves as a first node of the original pose description graph, each first node marks the human joint point type and the first included angle, and the bone connection between adjacent human joint points serves as a first edge, and each first edge marks the first connection length and the first coordinate point between adjacent first nodes.

[0065] Joint detection is realized based on a pre-trained deep learning pose estimation model. When the original image is downloaded from the cloud, the system inputs it into the pose parsing model to automatically detect the 17 standard joints of the human body in the image, including the tip of the nose, the neck, the left and right shoulders, the elbows, the wrists, the hips, the knees, the ankles, etc., and outputs the two-dimensional coordinates of each joint in the image coordinate system. Based on the detection results, the system constructs an original pose description graph: each joint is taken as the first node in the graph, the node type is identified by its semantic label (such as left shoulder, right knee), and the first edge is formed between adjacent joints through the skeleton to form the topological structure graph of the human body. Each first edge not only represents the connection relationship, but also carries the connection length between the two adjacent nodes (calculated by the Euclidean distance).

[0066] The conversion submodule is configured to generate a standard pose description graph based on the character pose key point coordinates of the target template; the standard pose description graph includes a plurality of second nodes and second edges, wherein each character pose key point is taken as a second node of the standard pose description graph, each second node is a character pose key point type, a skeleton connection between adjacent character pose key points is taken as a second edge, and each second edge is marked with a second included angle between adjacent second nodes, a second connection length of the second node, and a second coordinate point.

[0067] Each preset key point is taken as a second node in the graph; a second edge is formed between adjacent key points through a skeleton connection relationship to construct a standard topological structure of the human body. Each second edge not only represents the connection relationship, but also carries the connection length between the two adjacent nodes.

[0068] The first included angle and the second included angle are respectively calculated based on the coordinate points (first coordinate point / second coordinate point) of three consecutive joints and key points using vector cross product, thereby reflecting the bending degree or direction of the joint.

[0069] The alignment submodule is configured to perform alignment inference based on the graph structures of the original pose description graph and the standard pose description graph to obtain an optimal mapping relationship between the first nodes and the second nodes.

[0070] The alignment submodule includes the following units:

[0071] The neighborhood construction unit is configured to extract all adjacent nodes and adjacent edges directly connected based on the first nodes, and construct a first-order neighborhood structure based on the adjacent nodes and the adjacent edges.

[0072] Assume that the embodiment is processing a human standing pose of an original image, which is composed of multiple joints, such as head, shoulder, elbow, wrist, hip, knee, and ankle, etc. Each joint is taken as a first node and connected to other joints through a skeletal model, forming a simplified skeletal representation of a human body. When a specific first node, such as the right shoulder, is selected as a starting point, the neighborhood construction unit starts working:

[0073] First, all the first nodes directly connected to the right shoulder are identified: neck, right elbow, and top of the spine. Then, the adjacency edge information between these adjacent nodes is calculated and recorded, including the length and angle of each edge. For example, the edge from the right shoulder to the right elbow not only represents a part of the arm, but also carries the actual length and angle information of this limb.

[0074] Based on the information of these adjacent nodes and edges, a first-order neighborhood structure is constructed. This structure is essentially a local network with the selected first node as the center, surrounded by all adjacent nodes directly connected to it, and connected through specific edges to form a complete small skeletal segment. This first-order neighborhood structure helps to understand the local pose features, such as the position, direction, and relationship between limbs, providing basic data support for further pose adjustment and optimization.

[0075] The neighborhood feature construction unit is configured to construct a neighborhood feature based on the first node, the adjacent nodes, and the adjacent edges.

[0076] The human joint type of the first node (such as shoulder), two-dimensional coordinates, and the connection relationship between each adjacent node are obtained; then, for each adjacent edge, the connection length and the first angle are extracted. The embodiment encodes these geometric and topological information into a fixed-dimensional multi-dimensional vector as the neighborhood feature representation of the node. The vector contains the following elements: the center node type (the first node, represented by one-hot encoding), the normalized first connection length of each adjacent edge, the first angle, the type code of the adjacent node, etc.

[0077] The matching unit is configured to generate a similarity based on each neighborhood feature in the original pose description graph and the standard pose description graph, take the similarity as a probability, and construct a matching matrix according to the probability.

[0078] The neighborhood features of each first node are extracted from the original pose graph, and the corresponding neighborhood features of each second node are obtained from the standard pose graph. Then, the feature similarity between each pair of first and second nodes is calculated, and the cosine similarity is used as the measurement method: the dot product of the two neighborhood feature vectors is calculated and divided by the product of their lengths, resulting in a similarity value between -1 and 1, with a value closer to 1 indicating that the two nodes are more similar in geometric structure and topological relationship. All similarity values are arranged in rows to form an initial matching matrix of MxN (M is the number of nodes in the original graph, and N is the number of nodes in the standard graph), and each element in the matrix represents the matching probability of a certain first node and a certain second node.

[0079] An updating unit is configured to update the matching matrix for multiple rounds to generate a matching update matrix, and perform convergence evaluation based on the matching update matrix after each round of updating until the maximum number of updates is reached, and determine the optimal mapping relationship based on the final matching update matrix.

[0080] Although the initial stage calculates the similarity based on the neighborhood features of the same type label to establish the initial matching matrix, the direct matching method based on the type label has significant limitations and cannot cope with the complex human pose changes and structural diversity in real scenarios. For example, in some non-standard poses (such as crossed arms and bent sides), nodes of the same semantic type may have large differences in local geometric features and templates due to occlusion, deformation, or detection errors, which may cause different class nodes (such as "original right elbow" and "template left shoulder") to exhibit high apparent similarity in local structure, leading to false matching. It is difficult to distinguish such high-similarity but semantically incorrect matching pairs, and it is easy to fall into a local optimal trap. In addition, the rationality of human pose depends not only on the consistency of individual node types, but also on the coordination of the overall topological structure. Therefore, it is necessary to dynamically adjust the matching probability through a multi-round iteration mechanism, introduce context information and global structure verification, gradually eliminate inconsistent candidate matches, and enhance the confidence of correct matches. The iteration process allows the matching state of all nodes to be re-evaluated in each round, and the results of the previous round are fed back for optimization, thereby realizing the evolution from "local high similarity priority" to "global structure optimization". Only after multiple rounds of updating and convergence judgment can the final mapping relationship be ensured to meet the semantic correctness, geometric rationality, and visual naturalness, effectively improving the accuracy and robustness of pose alignment.

[0081] wherein the matching update matrix is obtained through a pre-trained graph message passing network model, and the matching matrix generated by the matching unit is the initial matching matrix. The initial matching matrix is iteratively updated based on the following convergence evaluation results.

[0082] Convergence evaluation is performed after each round of updating, including:

[0083] determine a Frobenius norm difference value based on the current matching matrix and the matching matrix of the previous round; if the Frobenius norm difference value is greater than or equal to a first target value, perform the next round of updating based on the current matching matrix.

[0084] The expression of the Frobenius norm difference value is:

[0085] ;

[0086] is the Frobenius norm difference value, M （t） is the matching matrix of the tth round, M （t-1） is the matching matrix of the (t-1) th round, is the Frobenius norm, is the probability of the i th row and the j th column in the matching matrix of the tth round, is the probability of the i th row and the j th column in the matching matrix of the (t-1) th round, p is the total number of rows of the matching matrix, and q is the total number of columns of the matching matrix.

[0087] Since each element in the matching matrix represents similarity or probability, the value is normalized between 0 and 1, so the change amplitude of the corresponding position between the adjacent two matching matrices is limited. In the initial iteration stage of the system, the matching relationship is not stable, and the overall change of the matrix is large, so the difference value is usually at a high level; as the iteration proceeds, the matching relationship between the nodes is gradually optimized, the difference value rapidly decreases and tends to be flat. Actual tests show that, in the human joint matching scene, when the difference value decreases to below 0.05, the overall structure of the matching matrix has basically converged, and only slight fluctuations of individual non-key nodes exist, and the improvement of the final matching result by continuing iteration is extremely limited, and may cause local shock and affect the efficiency. Therefore, the first target value is set to 0.05, on the one hand, the value is small enough to ensure that the system reaches a high-precision stable state in the pose alignment process, avoiding mismatch of key points due to premature termination; on the other hand, the value is not too strict, which can effectively control the calculation time on the premise of ensuring the matching quality, and meet the demand of the intelligent photo system for real-time response. At the same time, the value can be adaptively adjusted according to the body shape characteristics and pose complexity of different population categories (such as middle-aged men and small girls).

[0088] If the Frobenius norm difference value is less than the first target value, determine the second node with the highest probability of the first node in the matching matrix and mark it as a second sub-node, and determine the first node with the highest probability of the second node in the matching matrix and mark it as a first sub-node.

[0089] When the Frobenius norm difference value is less than the first target value, it indicates that the matching matrix has tended to be stable and enters the extraction stage of candidate matching pairs. At this time, each row in the matching matrix is traversed, and for each first node in the original pose description graph, the second node with the highest matching probability in the standard pose description graph is found, which is marked as the second child node of the first node. For example, if the original right shoulder has the highest matching value with the template right shoulder position in the matching matrix, the template right shoulder is determined as the second child node thereof. Conversely, it is further checked whether the second node also has the first node as the highest matching source in the matching matrix, i.e., whether the matching probability of the first node in the corresponding column is the maximum value. If so, the first node is marked as the first child node of the second node, forming a bidirectional optimal matching relationship.

[0090] Based on the first child node, a corresponding adjacent edge is determined, a first connection length is determined according to the first child node and the corresponding adjacent edge, and a first included angle is determined based on the first child node. Based on the second child node, a corresponding adjacent edge is determined, a second connection length is determined according to the second child node and the corresponding adjacent edge, and a second included angle is determined based on the second child node.

[0091] For a certain first child node (such as the right elbow) in the original pose description graph, the system first determines all adjacent edges directly connected thereto, such as the right shoulder-right elbow and right elbow-right wrist edges. Based on the end point coordinates of the two edges, the system calculates the Euclidean distance of each edge as the first connection length, which is used to represent the actual size of the corresponding limb segment in the original image, such as the length of the upper arm and forearm. At the same time, the system takes the first child node as the vertex, regards the right shoulder-right elbow and right elbow-right wrist as two vectors, and calculates the included angle between them as the first included angle through the vector included angle formula, which reflects the bending degree of the elbow joint in the original pose. Correspondingly, in the standard pose description graph, for the matched second child node (such as “right elbow” in the template), the system also identifies the adjacent edges connected thereto, such as the template right shoulder-template right elbow and template right elbow-template right wrist, and calculates the corresponding second connection length (such as the length of the upper arm and forearm in the template) and the second included angle based on the preset coordinates of the key points in the template.

[0092] The included angle deviation is determined based on the difference between the first included angle and the second included angle.

[0093] The adjacent node directly connected to the first child node is determined according to the adjacent edge corresponding to the first child node, and the first connection ratio is generated based on the first connection edge of the first child node and the adjacent node. The adjacent node directly connected to the second child node is determined according to the adjacent edge corresponding to the second child node, the second connection ratio is generated based on the second connection edge of the second child node and the adjacent node, and the connection ratio difference value is generated based on the first connection ratio and the second connection ratio.

[0094] For a first sub-node (e.g. right elbow) in the original pose graph, the directly connected adjacent nodes (e.g. right shoulder and right wrist) are determined according to the adjacent edges connected thereto. Then, the length of the edge from the right shoulder to the right elbow is calculated as the first connection length, and a ratio operation is performed between the first connection length and the total length of the limb segment (e.g. the distance from the right shoulder to the right wrist) to generate the first connection ratio, which represents the relative position of the right elbow in the overall structure of the upper arm and forearm. Similarly, in the standard pose graph, for the matched second sub-node (e.g. the right elbow in the template), the connected adjacent nodes (e.g. the template right shoulder and the template right wrist) are identified, the second connection length from the template right shoulder to the template right elbow is calculated, and a ratio operation is performed between the second connection length and the total length of the template limb segment (e.g. the distance from the template right shoulder to the template right wrist) to obtain the second connection ratio, which reflects the relative distribution of the joint in the template pose. The difference between the first connection ratio and the second connection ratio is taken to obtain the connection ratio difference value.

[0095] If the first sub-node and the second sub-node have the same character node type, and the angle deviation is less than a second target value and the connection ratio difference value is less than a third target value, it is determined that the first sub-node passes the structural consistency test, otherwise, the next round of update is performed based on the current matching matrix.

[0096] For the second target value, it is set according to the ergonomics standard and the common error range in the pose estimation task. Preferably, considering the actual limitations of human joint movement and the influence of different shooting angles on pose recognition, the second target value can be set to 15 degrees. The third target value depends on the understanding and acceptable range of the length ratio of different parts of the human body. Since there is a certain variability in the structure of the human body, but it generally follows a certain proportional relationship, the third target value is preferably between 0.2 and 0.3.

[0097] The same node type (e.g., both "right elbow" or "left knee") ensures correct correspondence at the semantic level, avoiding cross-part errors (e.g., mistaking a shoulder for a hip); secondly, the angle deviation is less than a second target value, indicating that the limb bending degree of the node in the original image is basically consistent with the target pose in the template, for example, the elbow bending angle of the original pose and the bending angle in the template are not much different, indicating that they have deformability in dynamic pose and can achieve natural transition through non-rigid transformation; thirdly, the connection proportion difference is less than a third target value, meaning that the relative position relationship (i.e., the length proportion of the limb segment) between the node and its adjacent node is coordinated in the original pose and the standard pose, for example, the length ratio of the upper arm to the forearm is close in the two poses, which conforms to the laws of human anatomy and prevents artifacts such as limb stretching or compression caused by proportion distortion. These three conditions jointly constitute a strict geometric constraint system, which not only verifies the rationality of the matching node in the topological structure, but also guarantees the visual naturalness and physical reasonableness of subsequent pose alignment and image deformation, so it can be determined that the first sub-node passes the structure consistency test and has the basic qualifications as a reliable mapping relationship.

[0098] The proportion of the first sub-nodes that pass the structure consistency test in the original pose description graph is counted, and if the proportion is less than 80%, the next round of update is performed based on the current matching matrix.

[0099] If 17 key nodes (such as head, shoulder, elbow, knee, etc.) are detected in the original pose graph, and after inspection, 14 first sub-nodes meet the conditions, the pass rate is 14 / 17 ≈ 82.4%; if only 13 pass, the proportion is 13 / 17 ≈ 76.5%. The system sets 80% as the determination threshold, if the proportion is less than 80%, it means that there are many inconsistent or unreliable matching pairs in the mapping relationship generated by the current matching matrix, reflecting that the pose alignment has not reached the ideal state, which may be due to initial matching errors, occlusion or too large pose difference, causing some key points to be not accurately matched. At this time, the system determines that the overall matching quality is insufficient, and needs to continue the next round of iteration update based on the current matching matrix.

[0100] Based on the first sub-node and the second sub-node, a candidate matching pair is determined; based on the candidate matching pair, a limb connection proportion change rate is determined; if the limb connection proportion change rate meets the target condition, it is determined that the current matching update matrix is in a convergent state, and the current matching update matrix is taken as the final matching update matrix; otherwise, the next round of update is performed based on the current matching matrix.

[0101] Based on each candidate matching pair, three consecutive nodes belonging to the same limb chain are identified, the first connection length and the second connection length of two segments of the limb chain in the original pose graph and the standard pose graph are respectively calculated, the relative proportions between the two segments of the limb in the original pose graph and the standard pose graph are calculated, the two relative proportions are subtracted, and the absolute value is taken to obtain the limb connection proportion change rate. If the limb connection proportion change rate is small, it indicates that the geometric structure of the two pose graphs in this local area is relatively consistent; otherwise, there may be a large deformation or error. For example, the distance from the shoulder to the left elbow is selected as the reference, and the limb connection proportion change rates of the remaining matching pairs relative to the reference are calculated. If all the calculated limb connection proportion change rates are less than or equal to the preset threshold (5%), it can be considered that the current matching update matrix has reached a convergent state, that is, the best matching relationship is found.

[0102] Once it is judged that the current matching update matrix meets the target condition and is in a convergent state, it is identified as the final matching update matrix. After a series of iterative optimization processes, a matrix that can best describe the corresponding relationship between the two pose graphs is obtained. This matrix not only reflects the accurate matching between the key points, but also reveals the transformation rules of the entire pose graph in space, including rotation, scaling and translation. However, if the limb connection proportion change rate calculated in a certain round does not meet the target condition, the next round of update needs to be continued according to the current matching matrix until a solution that meets the condition is found.

[0103] According to the probability of each candidate matching pair, the optimized matching pair is determined, and the Hungarian algorithm is performed on the optimized matching pair to obtain the optimal mapping relationship, that is, the one-to-one correspondence between the first sub-node and the second sub-node.

[0104] After completing the multiple rounds of iterative optimization and structure consistency inspection, the generation stage of the final mapping relationship is entered. The core goal of this stage is to establish a unique, conflict-free and globally optimal one-to-one correspondence between each first sub-node in the original pose and each second sub-node in the standard template. To this end, first, the candidate matching pairs in the current matching matrix are screened with high confidence, and a probability threshold of 0.5 is set to retain only the node pairs with a matching probability higher than the threshold. For example, the matching probability of the original right shoulder and the template right shoulder is 0.86, and it is retained; while the matching probability of the original left wrist and the template right knee is only 0.32, and it is excluded. Through this step, the optimized matching pair composed of high-confidence matching is constructed.

[0105] The candidate set is modeled as a weighted bipartite graph: the left vertex set is all first sub-nodes in the original pose (e.g. right elbow, left knee, etc.), the right vertex set is all second sub-nodes in the standard pose (e.g. template right elbow, template left knee, etc.), and each connecting edge represents a candidate matching relationship, with its weight being the corresponding matching probability. The Hungarian Algorithm is used to solve the maximum weighted bipartite graph matching. This algorithm can find the optimal solution that maximizes the total matching weight and satisfies the constraint that each node is matched only once in polynomial time, through systematic matrix transformation and augmented path search. Specifically, the algorithm evaluates all possible matching combinations, automatically excludes solutions with resource conflicts, and selects the highest overall confidence conflict-free matching path. For example, when there is a conflict between the original right elbow-template right elbow (0.86) and the original left wrist-template right elbow (0.79), the algorithm combines the matching conditions of other nodes to determine whether the original left wrist should be reassigned to the template left wrist to release the template right elbow to a more reasonable matching object, thereby achieving global optimal configuration. Finally, the Hungarian Algorithm outputs a set of accurate pairing results, i.e. the optimal mapping relationship, in which each first sub-node uniquely corresponds to a second sub-node, and each second sub-node is also matched by only one first sub-node.

[0106] The image processing submodule is used to generate a sparse point set according to the optimal mapping relationship; perform gradient analysis based on the original image to generate dynamic weights, and perform interpolation weighting on the sparse point set based on the dynamic weights to obtain a target point set; and map the target point set of the original image to the standard pose description graph to obtain an intermediate image.

[0107] According to the optimal mapping relationship, a sparse point set is generated; dynamic weights are generated based on gradient analysis of the original image, including:

[0108] Based on the original image, a semantic region is generated; the semantic region includes a skin region, a clothing region, and a background region.

[0109] The original image is input into a pre-trained deep learning semantic segmentation model to classify and predict each pixel in the image, outputting a pixel-level label map, and then dividing it into multiple semantic regions. Specifically, three core regions are identified and separated: skin region, clothing region, and background region. The skin region includes the face, neck, arms, legs, and other exposed skin parts. The model accurately extracts them based on skin color distribution, edge continuity, and human body topology priori. The clothing region covers shirts, pants, skirts, shoes, and other worn items. The system segments them based on texture, contour closure, and spatial association with joint points. The background region refers to the environment part other than the human body, such as walls, floors, props, etc. It is effectively stripped through foreground-background contrast analysis and edge isolation. All semantic regions are stored in the form of masks. Each mask is a binary image consistent with the resolution of the original image, which is used for subsequent application of differentiated image processing strategies in different regions.

[0110] Based on the original image, a multi-scale gradient intensity field is extracted, and based on the multi-scale gradient intensity field, a plurality of gradient maps are obtained.

[0111] Sobel, Scharr, or Gaussian derivative operators are used to calculate the gradient intensity of the image at different scales. Each scale corresponds to a different spatial resolution: small scales can capture fine points and texture details in the image, such as skin pores or clothing fabric lines; medium scales highlight medium-length edge structures, such as arm outlines or collar lines; large scales focus on overall shape and major outlines, such as human torso boundaries or background segmentation lines. By independently calculating the gradient amplitude at these scales, a gradient intensity field containing multiple scale information is constructed. Subsequently, based on the multi-scale gradient intensity field, a plurality of gradient maps corresponding to the gradient response maps at different scales are generated. Each gradient map takes the pixel-level gradient amplitude as the gray value, intuitively presenting the edge distribution at each scale. For example, the gradient map at a small scale is densely distributed with clothing wrinkles and facial fine textures, while the gradient map at a large scale clearly outlines the overall outline of the human body and the background boundary.

[0112] Based on the plurality of gradient maps, structure analysis is performed to obtain a structure importance map, and based on the structure importance map, a structure importance score for each pixel point is obtained.

[0113] The gradient field is analyzed to extract three key features: gradient direction consistency, which is the standard deviation of gradient direction within a local window, high consistency areas usually correspond to strong edges or contours (such as hairline, collar), low consistency may be texture or noise; gradient intensity contrast, which is the ratio of local gradient intensity to the surrounding area, high contrast indicates important edges, low contrast is secondary details; gradient connectivity, by analyzing the edge connection of the gradient map, long and continuous edges are identified as important structures (such as arm contour), while isolated short edges are considered as noise or texture. Finally, an important structure map is output, and each pixel value is the structure importance score, reflecting whether the position belongs to the structure edge.

[0114] According to the structure importance score of each semantic region, a semantic modulation factor is determined.

[0115] For each semantic region, the structure importance score distribution is analyzed, and a semantic modulation factor is assigned. For example, in the skin region, edges with high gradient intensity and strong direction consistency (such as facial contour, feature boundary) are detected, which are determined as real contours, and are given a higher semantic modulation factor (such as 0.9~1.0) to retain clear structural details; if local strong gradient but direction is chaotic, it may be noise or light artifact, and a lower semantic modulation factor (such as 0.3~0.5) is given to avoid distortion caused by excessive enhancement. In the clothing region, complex gradient structures such as wrinkles and patterns are allowed, so even if there are high gradient areas with inconsistent directions (such as sleeve wrinkles), they are considered as meaningful texture, and a medium-high semantic modulation factor (such as 0.7~0.9) is given to maintain the texture of the cloth; but if the gradient intensity is abnormally prominent and isolated, it may be a false edge, and a semantic modulation factor (0.25) is given. In the background region, the system expects the gradient to be smooth, with only significant changes at the edge of the foreground object. If isolated strong gradient points are detected in the background, they are likely to be noise or small object interference, and should be given a very low weight (such as 0.1~0.2) to prevent them from affecting the overall visual consistency.

[0116] According to the relationship between the semantic modulation factor and / or the structure importance score and the preset threshold condition, a dynamic weight map is generated, and the dynamic weight of the semantic region is determined based on the dynamic weight map.

[0117] The structural importance score and the corresponding semantic modulation factor of each pixel point are obtained. When the structural importance is greater than 0.8 and the semantic modulation factor is greater than 0.7, a dynamic weight of 0.9 is assigned, indicating that the region is a key contour (such as facial features, limb boundaries) and should be highly preserved; when the structural importance is greater than 0.5 and the semantic modulation factor is greater than 0.5, a dynamic weight of 0.6 is assigned, which is suitable for clothing wrinkles or secondary edges; and the rest is assigned a dynamic weight of 0.3, which is used to suppress noise or background interference. Finally, a dynamic weight map is obtained based on the dynamic weight of each pixel point.

[0118] For each first node of the original image, the dynamic weight of the node is obtained by looking up the corresponding dynamic weight map according to its first coordinate in the original image. These first nodes with dynamic weights are taken as input, and an interpolation algorithm (such as radial basis function (RBF) interpolation) is used for processing. In this interpolation method, each first node not only provides its coordinate information, but also carries the importance level determined by the dynamic weight. This means that when calculating the position of any point in the target point set, the degree of influence of the surrounding key points on it is no longer dependent on the geometric distance, but also considers the weight factor. For example, in RBF interpolation, the selection and parameter adjustment of the kernel function can make the first nodes with closer distance and larger dynamic weight have greater influence on the position of the newly generated point, thereby ensuring the stability and accuracy of the key parts in the deformation process.

[0119] After interpolation, a more dense and smooth target point set is obtained, which can more accurately describe the change path from the original pose to the target pose. Subsequently, a deformation mapping technique (such as thin plate spline (TPS)) is used to guide the conversion of the original image to the standard pose description graph according to the new point set. In this process, each pixel point in the original image will determine the final position according to the movement trajectory of the nearby first nodes, and this movement trajectory is calculated based on the weighted interpolation result described above. In this way, by comprehensively using dynamic weights, interpolation algorithms, and deformation mapping techniques, the original image can be smoothly and naturally mapped to the standard pose while maintaining the details and structure of the original image, thereby generating intermediate images.

[0120] The synthesis module performs layer superposition processing on the intermediate images to generate a synthesis image.

[0121] In the foregoing detailed description, reference is made to the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" etc. which describe a particular feature, structure, material, or characteristic in connection with at least one embodiment or example. The illustrative description in this specification is not necessarily to be taken in a literal sense. The use of the term "in one embodiment", "in some embodiments", "an example", "a specific example", or "some examples" etc. is not necessarily to be taken to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Moreover, the description of a particular feature, structure, material, or characteristic is not to be taken as limiting on the scope of the disclosure. Different embodiments or examples can be combined in different ways to form different embodiments or examples.

[0122] Furthermore, the terms "first", "second", etc. are used herein only to describe different features, structures, materials, etc. and do not imply relative importance or a listing of characteristics. Thus, a feature described as "first" can imply that the feature is a first feature in a listing of features, or that the feature is a first feature in a listing of features that includes the feature. In the description of the application, the meaning of "a plurality" is at least two, for example, two, three or four.

Claims

1. An intelligent photo system based on crowd classification, characterized in that, Includes the following modules: The instruction receiving module is used to receive user-input instructions on the population category. The photo template acquisition module is used to acquire a set of photo templates associated with the audience category instruction in response to the audience category instruction. The set of photo templates includes thumbnails composed of several photo templates. Each of the aforementioned photo templates includes: key points of the figure's pose and coordinates of the key points of the figure's pose; The target template acquisition module obtains the target template selected by the user based on multiple thumbnails in the photo template set; The acquisition module is used to acquire the user's raw images; The image analysis module is used to perform pose analysis based on the original image and the target template, generate an optimal mapping relationship, and generate an intermediate image based on the optimal mapping relationship. It includes the following sub-modules: The pose analysis submodule is used to perform human pose analysis on the original image to obtain human joints and human joint coordinates; and to construct an original pose description map based on the human joints and human joint coordinates. The original pose description map includes several first nodes and first edges, wherein each human joint is a first node of the original pose description map, each first node is marked with human joint type and first included angle, the skeletal connection between adjacent human joints is a first edge, and each first edge is marked with first connection length and first coordinate point between adjacent first nodes. The conversion submodule is used to generate a standard posture description map based on the coordinates of the key points of the character posture in the target template. The standard posture description map includes multiple second nodes and second sides, wherein each key point of the character posture is a second node of the standard posture description map, each second node is marked with the type of character joint and the second angle, the skeletal connection between adjacent key points of the character posture is a second side, and each second side is marked with the second connection length and the second coordinate point between adjacent second nodes. The alignment submodule, used for performing alignment inference based on the graph structure of the original attitude description graph and the standard attitude description graph to obtain the optimal mapping relationship between the first node and the second node, includes the following units: The neighborhood construction unit is used to extract all directly connected adjacent nodes and adjacent edges based on the first node, and to construct a first-order neighborhood structure based on the adjacent nodes and adjacent edges. The neighborhood feature construction unit is used to construct neighborhood features based on the first node, the adjacent nodes, and the adjacent edges; A matching unit is configured to generate a similarity based on each neighborhood feature in the original pose description map and the standard pose description map, use the similarity as a probability, and construct a matching matrix based on the probability; An update unit is used to update the matching matrix in multiple rounds to generate a matching update matrix; and to perform convergence evaluation based on the matching update matrix after each round of updates until the maximum number of update rounds is reached, and then determine the optimal mapping relationship based on the final matching update matrix. The image processing submodule is used to generate a sparse point set according to the optimal mapping relationship; perform gradient analysis based on the original image to generate dynamic weights; perform interpolation weighting on the sparse point set based on the dynamic weights to obtain a target point set; and map the target point set of the original image to the standard pose description map to obtain an intermediate image. The compositing module performs layer overlay processing on the intermediate image to generate a composite image.

2. The intelligent photo system based on crowd classification according to claim 1, characterized in that, A convergence evaluation is performed after each round of updates, including: Based on the current matching matrix and the matching matrix of the previous round, determine the difference in Frobenius norm; If the Frobenius norm difference is greater than or equal to the first target value, the next round of updates is performed based on the current matching matrix; If the Frobenius norm difference is less than the first target value, the first node is determined to be the second node with the highest probability in the matching matrix and marked as the second child node; the second node is determined to be the first node with the highest probability in the matching matrix and marked as the first child node. Based on the first child node, determine the corresponding adjacent edge, determine the first connection length based on the first child node and the corresponding adjacent edge, and determine the first included angle based on the first child node; based on the second child node, determine the corresponding adjacent edge, determine the second connection length based on the second child node and the corresponding adjacent edge, and determine the second included angle based on the second child node; The included angle deviation is determined based on the difference between the first included angle and the second included angle; Based on the adjacent edges corresponding to the first child node, determine the adjacent child nodes directly connected to the first child node, and generate a first connection ratio based on the first connection edges between the first child node and the adjacent child nodes; based on the adjacent edges corresponding to the second child node, determine the adjacent child nodes directly connected to the second child node, and generate a second connection ratio based on the second connection edges between the second child node and the adjacent child nodes. A connection ratio difference is generated based on the first connection ratio and the second connection ratio. If the first child node and the second child node have the same character node type, and the included angle deviation is less than the second target value and the connection ratio difference is less than the third target value, the first child node is determined to pass the structural consistency test; otherwise, the next round of updates is performed based on the current matching matrix. The percentage of the first child node that passes the structural consistency test in the original pose description graph is counted. If the percentage is less than 80%, the next round of updates is performed based on the current matching matrix. Candidate matching pairs are determined based on the first and second child nodes; Determine the rate of change of proportion based on candidate matching pairs; If the rate of change of the ratio meets the target condition, the current matching update matrix is ​​determined to be in a convergent state, and the current matching update matrix is ​​used as the final matching update matrix; otherwise, the next round of updates is performed based on the current matching matrix.

3. The intelligent photo system based on crowd classification according to claim 2, characterized in that, In the final matching update matrix, the optimal matching pair is determined based on the probability of each candidate matching pair. The Hungarian algorithm is then applied to the optimal matching pair to obtain the optimal mapping relationship, which is the one-to-one correspondence between the first child node and the second child node.

4. The intelligent photo system based on crowd classification according to claim 1, characterized in that, Based on the optimal mapping relationship, a sparse point set is generated; Based on the original image, gradient analysis is performed to generate dynamic weights, including: Based on the original image, a semantic region is generated; Based on the original image, a multi-scale gradient intensity field is extracted, and multiple gradient maps are obtained based on the multi-scale gradient intensity field. Structural analysis is performed based on multiple gradient maps to obtain a structural importance map, and a structural importance score is obtained for each pixel based on the structural importance map. The semantic modulation factor is determined based on the structural importance score of each semantic region; A dynamic weight map is generated based on the relationship between the semantic modulation factor and / or the structural importance score and a preset threshold condition, and the dynamic weight of the semantic region is determined based on the dynamic weight map.

5. The intelligent photo system based on crowd classification according to claim 4, characterized in that, The semantic region includes the skin region, the clothing region, and the background region.

Images