Model training method, mold penetration detection method, device and electronic equipment

Abstract

This application relates to the field of computer technology, and provides a model training method, a clipping detection method, an apparatus, and an electronic device. The method includes: acquiring multiple sample images and corresponding label information for each sample image, the label information including sample text information and clipping annotation information corresponding to the sample images; extracting features from the sample images using a visual encoder to obtain sample image features; encoding the sample text information using a text encoder to obtain text semantic features; performing clipping detection using a fusion output network based on the sample image features and text semantic features to obtain clipping detection results for the sample images; calculating a detection loss based on the clipping detection results and corresponding clipping annotation information; and adjusting the parameters of the clipping detection model based on the detection loss until the training termination condition is met. The clipping detection model trained by this application can be used to automatically identify clipping phenomena.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a model training method, a method for detecting clipping, a device, and an electronic device. Background Technology

[0002] In 3D models (such as virtual 3D models of game characters), especially when interactions between different 3D models are involved, errors in the 3D model settings may cause different 3D models to penetrate or overlap each other, resulting in clipping. Related technologies typically rely on manual observation of the 3D models to identify clipping, but this method is inefficient, especially when there are many 3D models. Summary of the Invention

[0003] In view of this, embodiments of this application propose a model training method, a clipping detection method, a device, and an electronic device to solve the problem of low efficiency in manually identifying clipping phenomena.

[0004] The embodiments of this application are implemented using the following technical solutions:

[0005] In a first aspect, embodiments of this application provide a model training method, comprising: acquiring multiple sample images and label information corresponding to each sample image, wherein the label information includes sample text information and clipping annotation information corresponding to the sample image, the clipping annotation information being used to indicate whether a 3D model presented in the sample image exhibits clipping; the sample text information includes at least one of sample image content description text and sample layout description text; performing feature extraction on the sample images by a visual encoder in a clipping detection model to obtain sample image features; performing text encoding on the sample text information by a text encoder in the clipping detection model to obtain text semantic features; performing clipping detection by a fusion output network in the clipping detection model based on the sample image features and the text semantic features to obtain a clipping detection result for the sample images; calculating a detection loss based on the clipping detection result of the sample images and the corresponding clipping annotation information; and adjusting the parameters of the clipping detection model based on the detection loss until the training termination condition is met.

[0006] Secondly, embodiments of this application provide a clipping detection method, comprising: acquiring a target image collected from a 3D model; performing clipping detection on the target image using a clipping detection model to obtain a clipping detection result for the target image, wherein the clipping detection model is trained according to the model training method described above.

[0007] Thirdly, embodiments of this application provide a model training apparatus, comprising: a first acquisition module, configured to acquire multiple sample images and label information corresponding to each sample image, wherein the label information includes sample text information and clipping annotation information corresponding to the sample image, and the clipping annotation information is used to indicate whether the three-dimensional model presented in the sample image has clipping phenomenon; the sample text information includes at least one of sample image content description text and sample layout description text of the sample image; a first feature extraction module, configured to extract features from the sample images by a visual encoder in a clipping detection model to obtain sample image features; a first text encoding module, configured to encode the sample text information by a text encoder in the clipping detection model to obtain text semantic features; a first clipping detection module, configured to perform clipping detection by a fusion output network in the clipping detection model based on the sample image features and the text semantic features to obtain a clipping detection result of the sample image; a loss determination module, configured to calculate a detection loss based on the clipping detection result of the sample image and the corresponding clipping annotation information; and an adjustment module, configured to adjust the parameters of the clipping detection model based on the detection loss until the training termination condition is met.

[0008] In some embodiments, the sample text information corresponding to the sample image includes sample layout description text, which indicates the key visual elements in the sample image and the pixel positions of the key visual elements in the sample image; the model training device further includes: a first key visual element detection module, used to perform key visual element detection on the sample image using an object detection model to obtain the detection result of the sample image, the detection result indicating the semantic category and corresponding pixel position information of each identified key visual element; and a sample layout description text generation module, used to generate the sample layout description text of the sample image based on the semantic category and corresponding pixel position information of the key visual elements in the detection result.

[0009] In some embodiments, the sample text information corresponding to the sample image includes sample image content description text; the model training device further includes: a first encoding module, used to encode the image features of the sample image to obtain the image encoding features of the sample image; and a first text decoding module, used to decode the image encoding features of the sample image to obtain the sample image content description text of the sample image.

[0010] In some embodiments, the clipping annotation information includes clipping labels and clipping location information. The clipping labels are used to indicate whether a 3D model presented in the sample image exhibits clipping, and the clipping location information is used to indicate the pixel location in the sample image where clipping exists. The clipping detection result includes clipping prediction labels and predicted clipping location information. The loss determination module includes: a first loss determination unit, used to calculate a clipping detection loss based on the clipping labels and the clipping prediction labels; a second loss determination unit, used to calculate a clipping location recognition loss based on the clipping location information and the predicted clipping location information; and a detection loss determination unit, used to weight the clipping detection loss and the clipping location recognition loss to obtain the detection loss.

[0011] In some embodiments, the fusion output network includes a feature fusion network and a classification network; the first clipping detection module includes: a fusion unit, used to fuse the sample image features and the text semantic features by the fusion network to obtain sample fusion features; and a first classification unit, used to classify the sample image according to the sample fusion features by the classification network to obtain the clipping detection result of the sample image.

[0012] In some embodiments, the fusion network includes a first fusion network and a residual fusion network; the fusion unit includes: a weighted fusion unit, used to perform weighted fusion of the sample image features and the text semantic features by the first fusion network to obtain reference sample fusion features; and a residual fusion processing unit, used to perform residual fusion processing on the reference sample fusion features by the residual fusion network to obtain the sample fusion features.

[0013] Fourthly, embodiments of this application provide a clipping detection device, comprising: a second acquisition module for acquiring an image to be detected from a 3D model; and a second clipping detection model for performing clipping detection on the image to be detected using the clipping detection model and outputting the clipping detection result of the image to be detected, wherein the clipping detection model is trained according to the model training method described above.

[0014] In some embodiments, the second clipping detection module includes: a first image feature extraction unit, configured to extract features from the image to be detected by a visual encoder in the clipping detection model to obtain image features; a first text encoding unit, configured to encode text information corresponding to the image to be detected by a text encoder in the clipping detection model to obtain text encoding features; the text information corresponding to the image to be detected includes at least one of image content description text and layout description text of the image to be detected; and a second classification unit, configured to classify the image to be detected based on the image features and the text encoding features by a fusion output network in the clipping detection model to obtain a clipping detection result of the image to be detected.

[0015] In other embodiments, the second clipping detection module includes: a second image feature extraction unit, used to extract features from the image to be detected by the visual encoder in the clipping detection model to obtain image features; and a third classification unit, used to classify the image based on the image features by the fusion output network in the clipping detection model to obtain the clipping detection result of the image to be detected.

[0016] In some embodiments, the image to be detected is captured by a virtual camera located in a virtual scene; the clipping detection result indicates that clipping exists in the image to be detected and indicates the corresponding clipping pixel position information; the clipping detection device further includes: a target mesh surface determination module, used to transform the clipping area represented by the clipping pixel position information to the world coordinate system corresponding to the virtual scene according to the camera parameters of the virtual camera, to determine the target mesh surface corresponding to the clipping pixel position information in the virtual scene; and a prompt information generation module, used to generate clipping prompt information according to the position information of the target mesh surface.

[0017] Fifthly, embodiments of this application provide an electronic device, including: a processor; a memory, wherein computer instructions are stored in the memory, and when the computer instructions are executed by the processor, the above-described model training method or through-model detection method is implemented.

[0018] Sixthly, embodiments of this application provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the aforementioned model training method or the model penetration detection method.

[0019] In a seventh aspect, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the aforementioned model training method or the model penetration detection method.

[0020] In this application, a clipping detection model is trained using sample images acquired from a 3D model, along with corresponding sample text information and clipping annotation information. The sample text information includes at least one of a sample image content description text and a sample layout description text. This text can guide the clipping detection model to understand the image content of the sample image, or to focus on pixel regions that require special attention. This allows the model to fine-tune its visual understanding of the sample image using the corresponding sample text information, or to dynamically analyze the sample image based on the textual instructions provided by the sample text information, resulting in a more targeted and focused understanding. This accelerates the learning of features in pixel regions where clipping occurs, improving the training efficiency of the clipping detection model. The clipping detection model trained in this application can automatically identify clipping phenomena from images without manual observation, thus significantly improving the efficiency of clipping detection.

[0021] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application.

[0024] Figure 2 This is a flowchart illustrating a model training method according to an embodiment of this application.

[0025] Figure 3 An exemplary schematic diagram of a sample image showing clipping in a 3D model is provided.

[0026] Figure 4 This is a schematic diagram of a residual cell according to an embodiment of this application.

[0027] Figure 5 This is a flowchart illustrating step 250 according to an embodiment of this application.

[0028] Figure 6This is a flowchart illustrating the sample layout description text for generating a sample image according to an embodiment of this application.

[0029] Figure 7 This is a flowchart illustrating the generation of sample image content description text according to an embodiment of this application.

[0030] Figure 8A This is a flowchart illustrating a model training method according to an embodiment of this application.

[0031] Figure 8B This is a flowchart illustrating the training of a penetration detection model according to an embodiment of this application.

[0032] Figure 9 This is a flowchart illustrating a method for detecting mold penetration according to an embodiment of this application.

[0033] Figure 10 This is a flowchart illustrating a penetration detection method according to another embodiment of this application.

[0034] Figure 11 This is a flowchart illustrating a penetration detection method according to another embodiment of this application.

[0035] Figure 12 This is a block diagram of a model training apparatus according to an embodiment of this application.

[0036] Figure 13 This is a block diagram of a penetration detection device according to an embodiment of this application.

[0037] Figure 14 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0038] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0039] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0040] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0041] In this document, "multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. In the following description, references to "some embodiments or some embodiment methods" describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0042] Figure 1 This is a schematic diagram illustrating an application scenario of this application according to an embodiment of this application, such as... Figure 1 As shown, this application scenario includes a training dataset 110 and an electronic device 120. The training dataset 110 includes training data for training a model to detect clipping, which includes multiple sample images, corresponding sample text information, and corresponding clipping annotation information. The electronic device 120 can obtain sample images, corresponding sample text information, and corresponding clipping annotation information from the training dataset 110. The model to be trained is deployed on the electronic device 120.

[0043] Sample images and their corresponding text information can be input into the clipping detection model. The visual encoder in the clipping detection model extracts features from the sample images to obtain sample image features. The text encoder in the clipping detection model encodes the text information to obtain text semantic features. The fusion output network in the clipping detection model performs clipping detection based on the sample image features and text semantic features to obtain the clipping detection result of the sample image. The detection loss is calculated based on the clipping detection result of the sample image and the corresponding clipping annotation information. The parameters of the clipping detection model are adjusted based on the detection loss until the training termination condition is met.

[0044] The training dataset 110 can be stored on electronic device 120 or other devices. Electronic device 120 can be a smartphone, tablet, laptop, desktop computer, smart TV, smartwatch, in-vehicle terminal, virtual interaction device, or a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms.

[0045] Figure 2 This is a flowchart illustrating a model training method according to an embodiment of this application, which can be executed by an electronic device, such as... Figure 2 As shown, it includes:

[0046] Step 210: Obtain multiple sample images and the corresponding label information for each sample image. The label information includes sample text information and clipping annotation information corresponding to the sample images. The clipping annotation information is used to indicate whether there is clipping in the 3D model presented in the sample image. The sample text information includes at least one of the sample image content description text and sample layout description text.

[0047] Sample images can be acquired from 3D models in virtual scenes. For example, images can be acquired from 3D game characters in a virtual game scene, or from 3D simulation models. The sample images present one or more 3D models, which can be 3D models of objects (such as virtual props, virtual clothing, virtual buildings, virtual rocks, virtual trees, etc.) or 3D models of people (such as game characters).

[0048] Multiple sample images include positive sample images and negative sample images. Positive sample images are those in which the presented 3D model exhibits clipping, while negative sample images are those in which the presented 3D model does not exhibit clipping. The ratio of positive to negative sample images can be set according to actual needs; for example, the ratio of positive sample images can be set to exceed that of negative sample images.

[0049] Clipping refers to the phenomenon where different 3D models can overlap or pass through each other. For example, there might be a clipping effect between the 3D models of game character A and B. When clipping occurs between different 3D models, such as 3D model M1 and 3D model M2, in a 2D image captured from M1 and M2, the pixel region containing 3D model M1 will contain pixels representing 3D model M2, and similarly, the pixel region containing 3D model M2 will contain pixels representing 3D model M1.

[0050] exist Figure 3 In the sample image shown, in the pixel region representing the three-dimensional model (let's call it the three-dimensional model M3) located in the middle, there are pixels in the pixel regions enclosed by the first dashed frame 310, the second dashed frame 320, and the third dashed frame 330 that clearly do not belong to the three-dimensional model M3. Therefore, it can be determined that there is clipping phenomenon in the regions corresponding to the first dashed frame 310, the second dashed frame 320, and the third dashed frame 330.

[0051] In some embodiments, the sample image content description text of the sample image may be used as the sample text information corresponding to the sample image, or the sample layout description text of the sample image may be used as the sample text information corresponding to the sample image, or the concatenated text obtained by concatenating the sample image content description text and the sample layout description text of the sample image may be used as the sample text information corresponding to the sample image.

[0052] The sample image content description text refers to the text used to describe the image content presented in the sample image. This text can describe the attributes of the 3D models presented in the sample image, as well as the relationships between different 3D models. Attributes of the 3D models include, for example, their color, movement, clothing, environment, number, held props, and semantic category (e.g., human, animal, tree). Examples of sample image description text include: "A dog is chasing a rabbit in the grass," or "Game character A is fighting with game character B while holding a sword."

[0053] In some embodiments, the sample image content description text may be a description text that mainly describes the three-dimensional model in the sample image that has a high probability of clipping.

[0054] In some embodiments, the sample image content description text can be manually written or generated by a graph-to-text model based on the sample image.

[0055] The sample layout description text of the sample image is used to indicate the key visual elements in the sample image and their pixel positions within the image. The pixel positions of the key visual elements in the sample image can be represented by the positions of the detection boxes surrounding the key visual elements.

[0056] Key visual elements in an image refer to visual elements that are of interest or focus in the image. The granularity of visual element division can be based on 3D models, where a single 3D model presented in the image is considered a visual element. For example, all 3D models presented in the sample image can be considered key visual elements. Alternatively, visual elements can be divided at the granularity of body parts, where a single body part can be considered a visual element. For example, all body parts on the 3D models in the sample image can be considered key visual elements.

[0057] In some embodiments, 3D models in an image that have a high probability of clipping can be considered key visual elements. Alternatively, body parts in an image that have a high probability of clipping can be considered key visual elements.

[0058] In some embodiments, the pixel positions of key visual elements in a sample image can be obtained through manual annotation. Alternatively, an object detection model can be used to identify key visual elements in the sample image to obtain their pixel positions.

[0059] In some embodiments, the names and pixel positions of key visual elements in a sample image can be combined into a statement according to a preset statement template to obtain a sample layout description text for the sample image. For example, the statement template could be: (XX) has a pixel position of (YY) in the image. The name of the key visual element is filled at position XX in the statement template, and the pixel position of the key visual element is filled at position YY in the statement template.

[0060] When a sample image contains multiple key visual elements, a statement template is used to combine each key visual element to obtain a sub-layout description text for each key visual element. The sub-layout description texts of all key visual elements in the sample image are then concatenated to obtain the sample layout description text of the sample image.

[0061] The name of a key visual element can also be referred to as its semantic category. In some embodiments, the names of key visual elements may be manually labeled. In other embodiments, the names of key visual elements may be identified through a neural network model.

[0062] The clipping annotation information may include clipping labels, which are used to indicate whether the 3D model presented in the sample image has clipping. The clipping labels include a first label (e.g., 1) to indicate the presence of clipping, and a second label (e.g., 0) to indicate the absence of clipping.

[0063] In some embodiments, the clipping annotation information may include clipping labels and clipping location information, whereby the clipping location information indicates the pixel locations in the sample image where clipping occurs. If clipping occurs in multiple pixel regions of the sample image, then the clipping location information of the sample image can indicate the locations of multiple pixel regions with clipping. The locations of the pixel regions with clipping can be characterized by the positions of the detection boxes surrounding the pixel regions with clipping. The clipping annotation information may be determined manually.

[0064] Step 220: The visual encoder in the penetration detection model extracts features from the sample image to obtain the sample image features.

[0065] A visual encoder is a neural network model used for feature extraction from images. Visual encoders can be constructed using convolutional neural networks, pooling neural networks, fully connected neural networks, etc. Examples of visual encoders include VisionTransformer (ViT, a visual encoder based on a transformer architecture) and NFNet (Normalizer-Free ResNets, a residual network architecture that does not require normalization), etc., without specific limitations here. A visual encoder can encode image features from sample images to obtain sample image features.

[0066] The process of a visual encoder encoding a sample image can be represented as follows:

[0067] Fimg=G(I); (Formula 1)

[0068] Where Fimg is the sample image feature of the sample image, G represents the visual encoder, and I represents the input sample image.

[0069] Step 230: The text encoder in the penetration detection model performs text encoding on the sample text information to obtain text semantic features.

[0070] A text encoder is a neural network model used to encode features from text. Examples of text encoders include BERT (Bidirectional Encoder Representations from Transformers), but no specific limitation is made here. The process involves first converting the sample text information into corresponding embedding vectors, which can be represented as:

[0071] E = Embed(T0); (Formula 2)

[0072] Where E is the embedding vector of the sample text information, Embed is the embedding function, and T0 is the sample text information.

[0073] Next, the embedding vector of the sample text information is input into the text encoder for text encoding. This process can be represented as:

[0074] V = W * E + b; (Formula 3)

[0075] V is the text semantic feature of the sample text information, W is the weight matrix of the text encoder, and b is the bias parameter of the text encoder. W and b can be determined through training.

[0076] A text encoder can encode the sample text information corresponding to a sample image to obtain text semantic features, which can reflect the semantics of the sample text information.

[0077] Step 240: The fusion output network in the clipping detection model performs clipping detection based on the sample image features and text semantic features to obtain the clipping detection result of the sample image.

[0078] The fusion output network can first fuse the features of the sample image and the semantic features of the text, and then perform clipping detection based on the fused features to obtain the clipping detection result of the sample image.

[0079] In some embodiments, the sample image features and text semantic features can be concatenated to achieve fusion, or the sample image features and text semantic features can be weighted to achieve fusion.

[0080] In some embodiments, the fusion output network includes a feature fusion network and a classification network; step 240 includes the following ①-②:

[0081] ① The fusion network fuses the sample image features and text semantic features to obtain the sample fusion features.

[0082] In some embodiments, the fusion network may include a multi-layer neural network (e.g., a fully connected network, a convolutional neural network, etc.) that can fuse sample image features with text semantic features at multiple levels.

[0083] In some embodiments, considering that the sample text information corresponding to the sample image is used to assist in understanding the sample image, to avoid the difference between the text feature space and the image feature space affecting the feature fusion effect, the text semantic features can be transformed to the image feature space before fusing the sample image features and text semantic features. The transformed text semantic features are represented in the image feature space and have the same dimension as the sample image features. After transforming the text semantic features to the image feature space, it is easier to interact and fuse different features (i.e., the transformed text semantic features and sample image features) in the same feature space (image feature space). Then, the transformed text semantic features are fused with the sample image features. In this case, the fusion output network also includes a feature alignment network, which is used to transform the text semantic features to the image feature space.

[0084] In some embodiments, the fusion network includes a first fusion network and a residual fusion network; the first fusion network can perform weighted fusion of sample image features and text semantic features to obtain reference sample fusion features; then, the residual fusion network performs residual fusion processing on the reference sample fusion features to obtain sample fusion features.

[0085] The weighted fusion process of the first fusion network can be represented as:

[0086] Ffusion=α*Fimg+β*Etxt; (Formula 4)

[0087] Here, Ffusion refers to the fusion features of the reference samples, Fimg refers to the sample image features, and Etxt can refer to textual semantic features or features obtained by transforming textual semantic features into the image feature space; α and β are weighted fusion coefficients. α and β can be adaptively determined by the first fusion network, meaning that the first fusion network can automatically adjust the values ​​of α and β for different sample images, where both α and β are non-negative numbers.

[0088] Residual fusion networks consist of multiple cascaded neural network layers. Furthermore, residual fusion networks also include skip connections between non-adjacent neural network layers, also known as residual connections. Adding residual connections can prevent overfitting and improve the generalization ability of the pattern detection model. For ease of description, a multi-layer neural network involving a residual connection is called a residual unit; correspondingly, a residual fusion network includes at least one residual unit.

[0089] Figure 4 An exemplary schematic diagram of a residual cell is shown, such as... Figure 4As shown, assuming the input to the first neural network layer is X, after the input X is processed by the first and second neural network layers, the output of the second neural network layer is F(X). In addition, the input X skips the first and second neural network layers and is added to the output F(X) of the second neural network layer to obtain F(X)+X. F(X)+X is used as the input to the third neural network layer. That is, it is equivalent to the residual unit involving the residual connection between the input of the first neural network layer and the input of the third neural network layer. Figure 4 The residual connection shown skips two neural network layers (i.e., the first neural network layer and the second neural network layer), but is not limited to this. The number of neural network layers skipped by each residual connection can be set according to actual needs, and no specific limit is made here.

[0090] In some embodiments, the residual fusion network may be Transformer-UNet (a U-shaped network based on a transformer architecture), ResNet (Residual Neural Network), etc., without specific limitations.

[0091] The process of residual fusion processing by the residual fusion network can be represented as follows:

[0092] Ftuned=TuneNet(Ffusion); (Formula 5)

[0093] Here, Ftuned represents the sample fusion feature, and TuneNet represents the residual fusion network.

[0094] In the visual encoder, text encoder, and residual fusion network described above, attention mechanisms can be incorporated to process features. This approach, using attention mechanisms and residual connections, improves the understanding of image features and prevents overfitting. Furthermore, the introduction of multi-task learning of textual and image information, along with adaptive determination of summation weights during the weighted fusion stage, enhances the ability of the penetration detection model to handle multimodal data.

[0095] ② The classification network classifies the sample images based on the sample fusion features to obtain the clipping detection results of the sample images.

[0096] In some embodiments, if the detection task of the clipping detection model is to identify whether there is clipping in an image, the classification network can identify whether there is clipping in the sample image based on the sample fusion features, and the clipping detection result of the corresponding sample image is used to indicate whether there is clipping in the sample image.

[0097] In some embodiments, if the detection task of the clipping detection model is to identify whether clipping exists in an image and the pixel regions where clipping exists, the classification network will identify whether clipping exists in each pixel region of the sample image and the location information of the pixel regions where clipping exists based on the sample fusion features.

[0098] Step 250: Calculate the detection loss based on the clipping detection results of the sample image and the corresponding clipping annotation information.

[0099] In some embodiments, clipping annotation information includes clipping labels, which indicate whether a 3D model presented in a sample image exhibits clipping behavior. Clipping detection results include clipping prediction labels, which indicate whether clipping behavior exists in the sample image identified by the clipping detection model. Based on this, a loss function can be used to calculate the detection loss by combining the clipping labels of the sample image with the clipping prediction labels of the sample image. Loss functions include, for example, absolute value loss, cross-entropy loss, mean squared error loss, and Manhattan loss. The calculation of the detection loss can be expressed as:

[0100] L=Loss(O,Y); (Formula 6)

[0101] Where O is the clipping label corresponding to the sample image, Loss() represents the loss function, and Y is the clipping prediction label corresponding to the sample image.

[0102] The detection loss can be calculated on a per-sample-image basis, for example, by calculating the detection loss for each sample image according to Formula 6. Alternatively, the detection loss can be calculated on a batch basis, where a batch contains multiple sample images, and the detection loss for a batch is equal to the sum of the detection losses for all sample images in the same batch.

[0103] In other embodiments, the clipping annotation information includes clipping labels and clipping location information. The clipping labels indicate whether a 3D model presented in the sample image exhibits clipping. The clipping detection result includes a clipping prediction label and predicted clipping location information; the predicted clipping location information indicates the location of the identified pixel region exhibiting clipping. In this embodiment, such as... Figure 5 As shown, step 250 includes:

[0104] Step 510: Calculate the penetration detection loss based on the penetration label and the penetration prediction label.

[0105] In some embodiments, a sample image can correspond to a clipping label, and correspondingly, the clipping detection result also includes a clipping prediction label. The clipping detection loss can be obtained by calculating the loss using a first loss function based on the clipping label corresponding to the sample image and the clipping prediction label in the clipping detection result. Similarly, the first loss function can be an absolute value loss function, a cross-entropy loss function, a mean squared error loss function, a Manhattan loss function, etc., without specific limitations here.

[0106] For example, if the first loss function is the cross-entropy loss function, the penetration detection loss La can be calculated according to Formula 7:

[0107]

[0108] Where y represents the clipping label corresponding to the sample image; This represents the predicted label for the sample image.

[0109] In other embodiments, when the sample text information corresponding to the sample image includes sample layout description text, the clipping annotation information may include clipping labels corresponding to each key visual element in the sample image. Correspondingly, the clipping detection result includes clipping prediction labels corresponding to each key visual element. In this case, a reference detection loss corresponding to each key visual element in the sample image can be calculated using a first loss function based on the clipping labels and corresponding clipping prediction labels. Then, the reference detection losses corresponding to all key visual elements in the sample image are summed to obtain the clipping detection loss.

[0110] For example, if the first loss function is the cross-entropy loss function, the penetration detection loss La can be calculated according to Formula 8:

[0111]

[0112] Where yi represents the through-mapping label corresponding to the i-th key visual element in the sample image; The label represents the clipping prediction label corresponding to the i-th key visual element in the sample image; N is the total number of key visual elements in the sample image.

[0113] In the above embodiments, the clipping detection loss is calculated on a sample image basis. In other embodiments, the clipping detection loss can also be calculated on a batch basis, that is, the clipping detection loss calculated for the same batch of sample images is added together to obtain the clipping detection loss for a batch.

[0114] For example, if the first loss function is the cross-entropy loss function, the penetration detection loss La for each batch can be calculated according to Formula 9:

[0115]

[0116] Where yji represents the through-mapping label corresponding to the i-th key visual element in the j-th sample image; Let P represent the clipping prediction label corresponding to the i-th key visual element in the j-th sample image; P is the total number of sample images in a batch.

[0117] Step 520: Calculate the loss of the mold-through position recognition based on the mold-through position information and the predicted mold-through position information.

[0118] The loss for identifying the clipping location can be calculated using a second loss function, based on the clipping location information corresponding to the sample image and the corresponding predicted clipping location information. Examples of second loss functions include the absolute value loss function, the mean squared error loss function, the intersection-over-union (IoU) loss function, the Generalized Intersection over Union (GIOU) loss function, the Distance-IOU (DIOU) loss function, and the Complete-IOU (CIOU) loss function considering aspect ratio, etc., without specific limitations here.

[0119] Similarly, the loss for identifying the morphing location can be calculated on a per-sample-image basis. Alternatively, it can be calculated on a batch basis, where a batch contains multiple sample images. The loss for identifying the morphing location of a batch is equal to the sum of the losses for identifying the morphing location of all sample images in the same batch.

[0120] Step 530: Weight the loss from mold penetration detection and the loss from mold penetration location recognition to obtain the detection loss.

[0121] The weighting coefficients for the clipping detection loss and the clipping position recognition loss can be set according to actual needs, for example, both can be 0.5, or they can be different.

[0122] In the above embodiments, the clipping annotation information includes clipping labels and clipping location information. Clipping location information is introduced as supervision. In this way, the clipping detection model is trained by clipping detection loss and clipping location recognition loss, so that the clipping detection model can not only identify whether there is clipping in the 3D model presented in the image, but also accurately identify the location of clipping in the image. In this way, it is not necessary for the clipping detection model to output the clipping detection result of the image and then manually locate the location of the clipping.

[0123] Step 260: Adjust the parameters of the penetration detection model based on the detection loss until the training termination condition is met.

[0124] Gradient descent can be used to calculate the gradient of the detection loss relative to the parameters of the current penetration detection model, and then the parameters of the penetration detection model can be adjusted in the direction of decreasing the gradient. In some embodiments, the parameters of the visual encoder, text encoder, and fusion output network in the penetration detection model can be adjusted. In some embodiments, the visual encoder and text encoder in the penetration detection model can be pre-trained; therefore, in step 260, the parameters of the visual encoder and text encoder can be fixed, and the parameters of the fusion output network in the penetration detection model can be adjusted.

[0125] The training termination condition can be that the number of iterations of the penetration detection model reaches a threshold, the calculated detection loss converges, the penetration detection accuracy of the penetration detection model reaches an accuracy threshold, or a combination of at least two of these conditions.

[0126] In this application, a clipping detection model is trained using sample images acquired from a 3D model, along with corresponding sample text information and clipping annotation information. By introducing sample text information, which includes at least one of a sample image content description text and a sample layout description text, the clipping detection model can be guided to understand the image content of the sample image. In other words, it can be guided to focus on pixel regions within the sample image that require special attention. This allows the clipping detection model to use the sample text information to perform visual understanding fine-tuning of the sample image (also known as visual instruction fine-tuning), or to dynamically analyze the sample image based on the text instruction provided by the sample text information, enabling a more targeted and focused understanding of the sample image, such as emphasizing regions with a higher probability of clipping. Therefore, this accelerates the learning of features of pixel regions where clipping occurs, improving the training efficiency of the clipping detection model. The clipping detection model trained using this application can be used to automatically identify clipping phenomena from images without requiring manual observation, thus significantly improving the efficiency of clipping detection. Furthermore, the clipping detection model trained using this application is applicable to the identification of clipping phenomena in various 3D models, demonstrating a wide range of applicability.

[0127] In some embodiments, the sample text information corresponding to the sample image includes sample layout description text, which indicates the key visual elements in the sample image and the pixel positions of the key visual elements in the sample image; such as Figure 6 As shown, the method also includes:

[0128] Step 610: The sample image is subjected to key visual element detection by the target detection model to obtain the detection results of the sample image. The detection results indicate the semantic category and corresponding pixel position information of each key visual element identified.

[0129] An object detection model refers to a neural network model used to identify objects in an image. Object detection models can be constructed using convolutional neural networks, fully connected neural networks, pooling neural networks, etc. In this application, the object detection target of the object detection model is a key visual element. In some embodiments, the object detection model can be RetinaNet (a deep learning model for object detection based on a feature pyramid network), EfficientDet (an efficient deep learning model for object detection), Swim Transformer (a window separator network), etc., without specific limitations.

[0130] The semantic category of a key visual element can be its name, such as tree, rock, game character A, dog, rabbit, etc. The pixel position information of a key visual element can be represented by the position of the detection box surrounding the pixel region containing the key visual element. For example, the pixel position information of a key visual element can include the pixel coordinates of the center point of the detection box and the size of the detection box (e.g., width and height).

[0131] The object detection model includes an image feature extraction network and a detection head. The image feature extraction network in the object detection model can extract the sample image features of the sample image, and then input the sample image features of the sample image into the detection head. The detection head performs key visual element detection based on the sample image features of the sample image and outputs the detection result of the sample image.

[0132] In some embodiments, the target detection model includes a semantic category detection head, a center point detection head, and a bounding box size detection head. The semantic category detection head can input sample image features of a sample image to perform target semantic recognition based on these features, outputting the semantic category of each key visual element. The center point detection head can input these features to predict the center point position based on these features, outputting the center point coordinates of the bounding box corresponding to each key visual element. The bounding box size detection head can input the sample image features and the center point coordinates of the bounding box to perform bounding box size regression, outputting the size of the bounding box corresponding to each key visual element.

[0133] Step 620: Generate sample layout description text for the sample image based on the semantic category and corresponding pixel position information of the key visual elements in the detection results.

[0134] In some embodiments, as described above, the semantic category of key visual elements in a sample image and the pixel position of key visual elements can be combined into a statement according to a preset statement template to obtain a sample layout description text of the sample image.

[0135] In other embodiments, the semantic category of each key visual element can be combined with the corresponding pixel position information to obtain the combination corresponding to each key visual element. Then, the combinations corresponding to all key visual elements in the sample image are spliced ​​together to obtain the sample layout description text of the sample image.

[0136] For example, suppose the combination corresponding to key visual element 1 is (dog, P1, w1, h1), the combination corresponding to key visual element 2 is (cat, P2, w2, h2), and the combination corresponding to key visual element 3 is (tree, P3, w3, h3). These three combinations can be concatenated to obtain the following concatenation result: {dog, P1, w1, h1; cat, P2, w2, h2; tree, P3, w3, h3}. This concatenation result is used as the sample layout description text for the sample image. Here, P1 to P3 represent the center coordinates of the detection boxes corresponding to the key visual elements; w1 to w3 represent the widths of the detection boxes corresponding to the key visual elements; and h1 to h3 represent the heights of the detection boxes corresponding to the key visual elements.

[0137] To ensure the accuracy of the object detection model in identifying key visual elements, it is necessary to train the object detection model in advance using the first training data. This will enable the object detection model to learn the features of each key visual element in the current application scenario (such as a virtual game scene corresponding to a game application), thus ensuring that each key visual element can be accurately identified in the future.

[0138] Through the above embodiments, the target detection model is used to detect key visual objects in the sample image, determine the semantic category and corresponding pixel position information of each key visual element in the sample image, and combine the semantic category and corresponding pixel position information of the key visual elements in the sample image to generate sample layout description text of the sample image. This realizes the automatic generation of sample layout description text of sample images without the need for manual position labeling and semantic category labeling of key visual objects in the sample image, which can significantly reduce the time for collecting training data for the model to pass through the mold detection model.

[0139] In some embodiments, the sample text information corresponding to the sample image includes a description of the sample image content; such as... Figure 7 As shown, the method also includes:

[0140] Step 710: Perform image feature encoding on the sample image to obtain the image encoding features of the sample image.

[0141] The image features of the sample image can be obtained by encoding the image features of the sample image through the first image encoder in the graph-based text model. The image encoding features of the sample image can reflect the semantic content of the sample image.

[0142] The graph-to-text model is used to generate descriptive text for an input image. The graph-to-text model includes a first image encoder and a text decoder. The first image encoder can be constructed using convolutional neural networks, pooling neural networks, fully connected networks, etc. In some embodiments, the structure of the first image encoder can be the same as the structure of the visual encoder described above, or the same as the structure of the image feature extraction network in the object detection model described above. The text decoder can be constructed using recurrent neural networks, long short-term memory networks, etc.

[0143] Step 720: Decode the image encoding features of the sample image to obtain the sample image content description text.

[0144] In the graph-based text model, the text decoder can decode each character according to the image encoding features of the sample image. After outputting the first character, it can decode and output the next character according to the image encoding features of the sample image and the already decoded characters, until the stop identifier is output. The sequence of characters obtained according to the output order is the sample image content description text of the sample image.

[0145] In some embodiments, the graph-to-text model can also be guided to generate text using descriptive guidance text. This guidance text indicates the focus of the description. The graph-to-text model then generates text on the sample image according to the focus indicated by the guidance text, ensuring that the object described in the resulting sample image content description text is the focus indicated by the guidance text. For example, if the guidance text is "Please summarize the overall scene presented in the image," the guidance text requires the focus to be on the global scene; if the guidance text is "Please describe the dog in the image," the guidance text requires the focus to be on the dog in the image; if the guidance text is "Please describe the foreground object in the image," the guidance text requires the focus to be on the foreground object in the image.

[0146] When a graph-to-text model is guided to generate text by a descriptive guide text, semantic features of the descriptive guide text can be extracted to obtain text features of the descriptive guide text. Then, the text features of the descriptive guide text are fused with the image encoding features of the sample image to obtain image-text fused features. Subsequently, the image-text fused features are input into the text decoder in the graph-to-text model, and the text decoder performs text decoding based on the image-text fused features to output the sample image content description text of the sample image.

[0147] The above embodiments enable the automatic generation of sample image content description text without manual writing, which can significantly reduce the time required to collect training data for the penetration detection model.

[0148] Figure 8A This is a flowchart illustrating a model training method according to an embodiment of this application. In this embodiment, the sample text information corresponding to each sample image includes sample image content description text and sample layout description text. For example... Figure 8A As shown, the main steps include the following:

[0149] (1) Data collection;

[0150] This step primarily involves collecting images as positive samples (i.e., images exhibiting clipping artifacts) and images as negative samples (i.e., images where clipping artifacts are not present in the presented 3D model). For example, it's possible to set up collecting n1 images as positive samples and n2 images as negative samples. In a specific embodiment, image acquisition can be performed within a 3D virtual game scene.

[0151] Next, the acquired images are labeled. On one hand, clipping annotation is performed on the images to obtain clipping annotation information; on the other hand, sample image content description text and sample layout description text are generated. This text can be manually written to obtain the sample image content description text and sample layout description text; or it can be generated according to… Figure 6 and Figure 7 The corresponding implementation automatically generates sample image content description text and sample layout description text for the image. These texts, along with clipping annotation information, facilitate the clipping detection model's understanding of the image content.

[0152] (2) Data preprocessing;

[0153] Data preprocessing may include normalizing the image and embedding the sample image content description text and sample layout description text separately to obtain the corresponding embedding representation vectors.

[0154] The process of normalizing an image can be described by Equation 10:

[0155] Inorm = σI - μ; (Formula 10)

[0156] Where Inorm is the normalized image, I is the original image, μ is the image mean, and σ is the standard deviation.

[0157] In some embodiments, to enrich the training data and improve the generalization ability of the clipping detection model, the normalized image can be enhanced. The enhancement process can be described by Equation 11:

[0158] Iaug=Gaussian(Inorm); (Formula 11)

[0159] Iaug is the enhanced image, and Gaussian is the enhanced Gaussian function applied to the normalized image using Inorm.

[0160] Both the enhanced and normalized images can be used as sample images in this application. It is understood that the normalized image and the image obtained by enhancing the normalized image share the same label information (including clipping annotation information, sample image content description text, and sample layout description text).

[0161] In some embodiments, the acquired image may also be denoised (e.g., Gaussian noise may be randomly added), and the denoised image may also be used as a sample image in this application.

[0162] Furthermore, considering the need for periodic testing and validation of the clipping detection model during training, the dataset obtained through the above process (including multiple images and their corresponding labels) can be divided into a training set, a validation set, and a test set to facilitate the training and evaluation of the clipping detection model. The data in the training set is used to train the clipping detection model, the data in the test set is used to test the accuracy of the model in clipping detection, and the data in the validation set is used to verify the accuracy of the model in clipping detection. The formula for dataset partitioning is:

[0163] D = Dtrain ∪ Dval ∪ Dtest; (Formula 12)

[0164] |Dtrain|:|Dval|:|Dtest|=a:b:c;(Formula 13)

[0165] Where D represents all datasets, and Dtrain, Dval, and Dtest represent the training set, validation set, and test set, respectively. |Dtrain| represents the number of samples in the training set Dtrain, |Dval| represents the number of samples in the validation set Dval, and |Dtest| represents the number of samples in the test set Dtest. An image and its corresponding label are considered a sample. The values ​​of a, b, and c can be set according to actual needs; for example, a:b:c = 6:2:2. Typically, the training set has the largest number of samples, exceeding the number of samples in the validation set and the test set.

[0166] (3) Train the model for detecting clipping.

[0167] In some embodiments, the LLaVa model can be used as the clipping detection model in this application to train the LLaVa model to enable it to detect clipping.

[0168] Figure 8B This is a flowchart illustrating the training of a penetration detection model according to an embodiment of this application, as shown below. Figure 8B As shown, a graph-to-text model can generate sample image content description text for a sample image. A target detection model detects key visual elements in the sample image, obtaining detection results, and then generates sample layout description text based on these results. Next, the sample image, its sample image content description text, and its sample layout description text are input into a clipping detection model for clipping detection, outputting the clipping detection result. Then, based on the clipping detection result and the corresponding clipping annotation information of the sample image, the detection loss is calculated, and the parameters of the clipping detection model are adjusted according to the detection loss. Since both the visual dimension sample image and the text dimension sample image content description text and sample layout description text are input into the clipping detection model, the clipping detection model can combine visual-textual understanding for prediction. Specific implementation details of the above process can be found in the embodiments described above, and will not be repeated here.

[0169] During the training of the clipping detection model, it can be set that after every N iterations (N is a positive integer, which can be set according to actual needs, such as N being 5000, 8000, 10000, etc.), the clipping detection model is tested using a test set to test whether the clipping detection accuracy of the model meets the requirements. If it does not meet the requirements, the clipping detection model is trained again using data from the training set. If it meets the requirements, the clipping detection model can be validated using data from the validation set to verify whether the accuracy of the clipping detection model meets the required accuracy.

[0170] After training the clipping detection model according to the above process, the model can learn the features of pixel regions where clipping occurs, and thus identify whether a 3D model in an image has clipping. Once trained, the clipping detection model can be used to detect clipping in input images.

[0171] Figure 9 This is a flowchart illustrating a method for detecting penetration according to an embodiment of this application. This method can be executed by an electronic device, such as a terminal or server. Figure 9 As shown, it includes:

[0172] Step 910: Obtain the image to be detected from the 3D model.

[0173] The image to be detected may contain one or more 3D models. In some embodiments, the image to be detected can be acquired by capturing images of the 3D models in a 3D virtual scene. In some embodiments, the image to be detected can be acquired by capturing images of the 3D models in a 3D simulation environment (e.g., a building simulation environment).

[0174] For example, in virtual scenes built for game applications, a virtual camera is typically included. This virtual camera simulates the user's perspective when viewing the virtual scene and captures images to obtain the image to be detected. Since the final game screen presented to the user is also captured by the virtual camera, the image to be detected and the game screen seen by the user are from the same perspective. This is equivalent to performing clipping detection on the image from the user's perspective beforehand and repairing it in time, ensuring that there is no clipping in the game screen presented to the user.

[0175] Step 920: Perform clipping detection on the image to be detected using the clipping detection model to obtain the clipping detection result of the image to be detected. The clipping detection model is trained according to the model training method in any of the above embodiments.

[0176] Similarly, the clipping detection result of the image to be detected may include a clipping prediction label, which indicates whether clipping exists in the image. In some embodiments, the clipping detection result of the image to be detected may include a clipping prediction label and clipping pixel location information, which indicates the location of pixel regions in the image where clipping exists. It is understood that if the clipping prediction label indicates that there is no clipping in the image, the clipping pixel location information in the clipping detection result may be empty.

[0177] In some embodiments, step 920 includes: extracting features from the image to be detected by the visual encoder in the penetration detection model to obtain image features; and classifying the image based on the image features by the fusion output network in the penetration detection model to obtain the penetration detection result of the image to be detected.

[0178] In other words, after the clipping detection model has been trained, the text encoder can be removed, and the visual encoder and the fused output network can be used as the clipping detection model to perform clipping detection on the input image. In this case, it is not necessary to generate image content description text and / or layout description text for the image to be detected.

[0179] Since the clipping detection model has been pre-trained using multiple sample images and their corresponding text information, the text information can guide the model to focus on important objects within the images. During training, the model learns which objects in the input images require special attention (e.g., objects with a high probability of clipping). When used for clipping detection, the model can automatically prioritize these objects based on the features learned during training, ensuring accurate clipping detection. This allows for automatic clipping detection from images without requiring manual observation of the 3D model.

[0180] In other embodiments, such as Figure 10 As shown, step 920 includes:

[0181] Step 1010: The visual encoder in the through-hole detection model extracts features from the image to be detected to obtain image features.

[0182] Step 1020: The text encoder in the through-detection model performs text encoding on the text information corresponding to the image to be detected to obtain text encoding features; the text information corresponding to the image to be detected includes at least one of the image content description text and layout description text of the image to be detected.

[0183] Similarly, the image content description text or the layout description text of the image to be detected can be used as the text information corresponding to the image to be detected. Alternatively, the concatenated text obtained by concatenating the image content description text and the layout description text of the image to be detected can be used as the text information corresponding to the image to be detected.

[0184] The image content description text and layout description text of the image to be detected can be manually written.

[0185] In some embodiments, the layout description text of the image to be detected can be generated according to the following process: Key visual elements of the image to be detected are detected using an object detection model to obtain the detection result of the image to be detected. The detection result indicates the semantic category and corresponding pixel position information of each identified key visual element; based on the semantic category and corresponding pixel position information of the key visual elements in the detection result of the image to be detected, the layout description text of the image to be detected is generated. The specific implementation details of this process are as follows... Figure 6 The corresponding implementation is similar and will not be described in detail here.

[0186] In some embodiments, the image content description text of the image to be detected can be generated according to the following process: image feature encoding of the image to be detected to obtain the image encoded features of the image to be detected; and text decoding of the image encoded features of the image to be detected to obtain the image content description text of the image to be detected. The specific implementation details of this process are as follows... Figure 7 The corresponding implementation is similar and will not be described in detail here.

[0187] Step 1030: The fusion output network in the clipping detection model classifies the image based on image features and text encoding features to obtain the clipping detection result of the image to be detected.

[0188] As described above, the fusion output network includes a feature fusion network and a classification network; correspondingly, step 1030 includes: fusing image features and text encoding features by the fusion network to obtain target fusion features; and classifying the image to be detected by the classification network based on the target fusion features to obtain the clipping detection result of the image to be detected.

[0189] In some embodiments, the fusion output network includes a feature alignment network. Before the fusion network fuses the image features and text coding features to obtain the target fusion feature, step 1030 further includes: transforming the text coding features to the image feature space where the image features are located by the feature alignment network to obtain the transformed text coding features; correspondingly, the fusion network subsequently fuses the image features and the transformed text coding features to obtain the target fusion feature.

[0190] In some embodiments, the fusion network includes a first fusion network and a residual fusion network; the step of fusing image features and text coding features by the fusion network to obtain target fusion features may include: the first fusion network performing weighted fusion of image features and text coding features (or transformed text coding features) of the image to be detected to obtain first fusion features; and the residual fusion network performing residual fusion processing on the first fusion features to obtain target fusion features.

[0191] In the above embodiments, text information corresponding to the image to be detected is introduced to assist the clipping detection model in understanding the image content of the image to be detected, or to guide the clipping detection model to focus on pixel areas in the image that require special attention. These pixel areas may include, for example, the pixel areas containing the objects described in the image content description text, or the pixel areas containing the key visual elements indicated in the layout description text. This allows the clipping detection model to understand the image to be detected more accurately and to perform a more focused visual understanding, such as paying particular attention to pixel areas prone to clipping, thereby improving the accuracy of the obtained clipping detection results.

[0192] In some embodiments, the clipping detection result of the image to be detected may include clipping prediction labels and clipping pixel position information. The clipping detection result may also include a labeled image, which is an image obtained by marking the pixel regions in the image to be detected that exhibit clipping phenomena according to the pixel positions indicated by the clipping pixel position information. For example, a detection box is added to the image to be detected to surround the pixel regions exhibiting clipping phenomena, resulting in a labeled image. The position of the detection box in the labeled image is the position indicated by the clipping pixel position information. Based on this, after step 920, the method further includes: generating first prompt information based on the labeled image. The first prompt information may include the labeled image, so that the user (e.g., the creator of the 3D model) can quickly locate the location of the clipping phenomenon in the 3D model based on the labeled image in the first prompt information, so as to promptly repair the 3D model with clipping phenomena.

[0193] In some embodiments, the image to be detected is captured by a virtual camera located in a virtual scene; the clipping detection result indicates that clipping exists in the image to be detected, and indicates the corresponding clipping pixel position information; such as Figure 11 As shown, after step 910, the method further includes:

[0194] Step 1110: Based on the camera parameters of the virtual camera, transform the clipping region represented by the clipping pixel position information to the world coordinate system corresponding to the virtual scene to determine the target mesh surface corresponding to the clipping pixel position information in the virtual scene.

[0195] The camera parameters of a virtual camera include its intrinsic parameter matrix. Assuming the coordinates of the optical center of the virtual camera are (x0, y0) and the focal length of the virtual camera is f, the intrinsic parameter matrix of the virtual camera can be:

[0196]

[0197] Based on the camera parameters of the virtual camera and the depth values ​​of the pixels in the test image, the target transformation matrix for transforming the pixels in the test image to the world coordinate system corresponding to the virtual scene can be determined. For example, if the depth value of pixel P in the test image is depth, then the target transformation matrix K1 is:

[0198] K1 = defth * K- 1 ;(Formula 15)

[0199] Thus, after determining the target transformation matrix, the clipping region represented by the clipping pixel position information can be transformed into the world coordinate system corresponding to the virtual scene. This determines the position information of the target mesh surface corresponding to the clipping pixel position information in the virtual scene (i.e., the coordinates of each vertex on the mesh surface in the world coordinate system of the virtual scene), thereby determining the target mesh surface. This target mesh surface is the mesh surface on the 3D model where the clipping phenomenon is reversed.

[0200] In some embodiments, when acquiring the image to be detected in a virtual environment, a depth image is acquired from the same viewpoint. The depth image reflects the depth value corresponding to each pixel in the image to be detected. Based on this, after obtaining the clipping detection result, the coordinates of the center pixel in the pixel region indicated by the clipping pixel position information can be determined. Then, the depth value of the pixel represented by the coordinates of the center pixel is obtained in the depth image, and the target transformation matrix is ​​determined accordingly in the manner described above, and the target mesh surface is subsequently determined accordingly.

[0201] Step 1120: Generate clipping prompt information based on the position information of the target mesh surface.

[0202] The clipping warning information may include the location information of the target mesh face. In some embodiments, the target mesh face on the 3D model can be marked and displayed based on the clipping warning information, for example, the display style of the target mesh face may differ from the display samples of other mesh faces on the 3D model. This facilitates quick location of the mesh face exhibiting clipping after opening the 3D model, without requiring manual searching and location.

[0203] The clipping detection method provided in this application can be used to detect clipping in 3D models in game virtual scenes, so as to facilitate subsequent clipping repair of 3D models based on the clipping detection results. Since it does not require manual inspection and identification of clipping areas by observing the model, it can improve the efficiency of clipping repair and quickly optimize 3D models in game scenes.

[0204] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0205] Figure 12 This is a block diagram of a model training apparatus according to an embodiment of this application, such as... Figure 12As shown, the model training device includes: a first acquisition module 1210, used to acquire multiple sample images and corresponding label information for each sample image. The label information includes sample text information and clipping annotation information corresponding to the sample images. The clipping annotation information is used to indicate whether the 3D model presented in the sample image has clipping phenomena. The sample text information includes at least one of the sample image content description text and sample layout description text. A first feature extraction module 1220 is used to extract features from the sample images by the visual encoder in the clipping detection model to obtain sample image features. A first text encoding module 1230 is used to encode the sample text information by the text encoder in the clipping detection model to obtain text semantic features. A first clipping detection module 1240 is used to perform clipping detection by the fusion output network in the clipping detection model based on the sample image features and text semantic features to obtain the clipping detection result of the sample images. A loss determination module 1250 is used to calculate the detection loss based on the clipping detection result of the sample images and the corresponding clipping annotation information. An adjustment module 1260 is used to adjust the parameters of the clipping detection model based on the detection loss until the training termination condition is met.

[0206] In some embodiments, the sample text information corresponding to the sample image includes sample layout description text, which indicates the key visual elements in the sample image and the pixel positions of the key visual elements in the sample image; the model training device further includes: a first key visual element detection module, used to perform key visual element detection on the sample image through an object detection model to obtain the detection result of the sample image, the detection result indicating the semantic category and corresponding pixel position information of each identified key visual element; and a sample layout description text generation module, used to generate sample layout description text of the sample image based on the semantic category and corresponding pixel position information of the key visual elements in the detection result.

[0207] In some embodiments, the sample text information corresponding to the sample image includes sample image content description text; the model training device further includes: a first encoding module, used to encode the image features of the sample image to obtain the image encoding features of the sample image; and a first text decoding module, used to decode the image encoding features of the sample image to obtain the sample image content description text of the sample image.

[0208] In some embodiments, the clipping annotation information includes clipping labels and clipping location information. The clipping labels are used to indicate whether a 3D model presented in the sample image exhibits clipping, and the clipping location information is used to indicate the pixel location in the sample image where clipping exists. The clipping detection result includes clipping prediction labels and predicted clipping location information. The loss determination module 1250 includes: a first loss determination unit, used to calculate clipping detection loss based on the clipping labels and clipping prediction labels; a second loss determination unit, used to calculate clipping location recognition loss based on the clipping location information and predicted clipping location information; and a detection loss determination unit, used to perform weighted processing on the clipping detection loss and clipping location recognition loss to obtain a detection loss.

[0209] In some embodiments, the fusion output network includes a feature fusion network and a classification network; the first clipping detection module 1240 includes: a fusion unit, used to fuse sample image features and text semantic features by the fusion network to obtain sample fusion features; and a first classification unit, used to classify the sample image according to the sample fusion features by the classification network to obtain the clipping detection result of the sample image.

[0210] In some embodiments, the fusion network includes a first fusion network and a residual fusion network; the fusion unit includes: a weighted fusion unit, used to perform weighted fusion of sample image features and text semantic features by the first fusion network to obtain reference sample fusion features; and a residual fusion processing unit, used to perform residual fusion processing on the reference sample fusion features by the residual fusion network to obtain sample fusion features.

[0211] Figure 13 This is a block diagram of a penetration detection device according to an embodiment of this application, such as... Figure 13 As shown, the clipping detection device includes: a second acquisition module 1310, used to acquire the image to be detected from the 3D model; and a second clipping detection module 1320, used to perform clipping detection on the image to be detected through a clipping detection model to obtain the clipping detection result of the image to be detected. The clipping detection model is trained according to the model training method described above.

[0212] In some embodiments, the second clipping detection module 1320 includes: a first image feature extraction unit, used to extract features from the image to be detected by a visual encoder in the clipping detection model to obtain image features; a first text encoding unit, used to encode text information corresponding to the image to be detected by a text encoder in the clipping detection model to obtain text encoding features; the text information corresponding to the image to be detected includes at least one of image content description text and layout description text of the image to be detected; and a second classification unit, used to classify the image to be detected by a fusion output network in the clipping detection model according to the image features and text encoding features to obtain the clipping detection result of the image to be detected.

[0213] In other embodiments, the second clipping detection module 1320 includes: a second image feature extraction unit, used to extract features from the image to be detected by the visual encoder in the clipping detection model to obtain image features; and a third classification unit, used to classify the image based on the image features by the fusion output network in the clipping detection model to obtain the clipping detection result of the image to be detected.

[0214] In some embodiments, the image to be detected is captured by a virtual camera located in a virtual scene; the clipping detection result indicates that clipping exists in the image to be detected and indicates the corresponding clipping pixel position information; the clipping detection device further includes: a target mesh surface determination module, used to transform the clipping area represented by the clipping pixel position information to the world coordinate system corresponding to the virtual scene according to the camera parameters of the virtual camera, and determine the target mesh surface corresponding to the clipping pixel position information in the virtual scene; and a prompt information generation module, used to generate clipping prompt information according to the position information of the target mesh surface.

[0215] Figure 14 A schematic diagram of a computer system suitable for implementing the electronic device of the embodiments of this application is shown. This electronic device is used to implement the model training method and the clipping detection method provided in this application. It should be noted that... Figure 14 The computer system 1400 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0216] like Figure 14 As shown, the computer system 1400 includes a Central Processing Unit (CPU) 1401, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1402 or programs loaded from storage portion 1408 into Random Access Memory (RAM) 1403. The RAM 1403 also stores various programs and data required for system operation. The CPU 1401, ROM 1402, and RAM 1403 are interconnected via a bus 1404. An Input / Output (I / O) interface 1405 is also connected to the bus 1404.

[0217] The following components are connected to I / O interface 1405: an input section 1406 including a keyboard, mouse, microphone, etc.; an output section 1407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to I / O interface 1405 as needed. Removable media 1411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1410 as needed so that computer instructions read from it can be loaded into storage section 1408 as needed.

[0218] In particular, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising computer instructions. When these computer instructions are executed by the central processing unit (CPU) 1401, various functions defined in the system of this application are performed.

[0219] This application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods described in any of the above method embodiments.

[0220] It should be noted that the computer-readable storage medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0221] In the embodiments of this application, the terms "module" or "unit" refer to computer instructions or a portion of computer instructions that have a predetermined function and work together with other related parts to achieve a predetermined goal. These instructions can be implemented, wholly or partially, using software, hardware (e.g., processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that functions as a whole.

[0222] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A model training method, characterized in that, include: Multiple sample images and corresponding label information for each sample image are obtained. The label information includes sample text information and clipping annotation information corresponding to the sample image. The clipping annotation information is used to indicate whether the 3D model presented in the sample image has clipping phenomenon. The sample text information includes at least one of the sample image content description text and sample layout description text. The visual encoder in the penetration detection model extracts features from the sample image to obtain the sample image features; The text encoder in the penetration detection model performs text encoding on the sample text information to obtain text semantic features; The fusion output network in the model performs clipping detection based on the sample image features and the text semantic features to obtain the clipping detection result of the sample image. The detection loss is calculated based on the penetration detection results of the sample images and the corresponding penetration annotation information; Based on the detection loss, adjust the parameters of the penetration detection model until the training termination condition is met.

2. The method according to claim 1, characterized in that, The sample text information corresponding to the sample image includes sample layout description text, which indicates the key visual elements in the sample image and the pixel positions of the key visual elements in the sample image. The method further includes: The sample image is subjected to key visual element detection by an object detection model to obtain the detection result of the sample image. The detection result indicates the semantic category and corresponding pixel position information of each identified key visual element. Based on the semantic categories and corresponding pixel position information of key visual elements in the detection results, a sample layout description text for the sample image is generated.

3. The method according to claim 1, characterized in that, The sample text information corresponding to the sample image includes a description of the sample image content; the method further includes: The sample image is subjected to image feature encoding to obtain the image encoded features of the sample image; The image encoding features of the sample image are decoded to obtain the sample image content description text.

4. The method according to claim 1, characterized in that, The clipping annotation information includes clipping labels and clipping location information. The clipping labels are used to indicate whether there is clipping in the 3D model presented in the sample image, and the clipping location information is used to indicate the pixel location in the sample image where clipping exists. The clipping detection result includes clipping prediction labels and predicted clipping location information. The step of calculating the detection loss based on the penetration detection results of the sample image and the corresponding penetration annotation information includes: Calculate the penetration detection loss based on the penetration label and the penetration prediction label; Calculate the mold position recognition loss based on the mold position information and the predicted mold position information; The detection loss is obtained by weighting the loss of mold penetration detection and the loss of mold penetration position recognition.

5. The method according to any one of claims 1 to 4, characterized in that, The fusion output network includes a feature fusion network and a classification network; The step of performing clipping detection by the fusion output network in the clipping detection model based on the sample image features and the text semantic features to obtain the clipping detection result of the sample image includes: The fusion network fuses the sample image features and the text semantic features to obtain sample fusion features; The classification network performs classification based on the sample fusion features to obtain the clipping detection result of the sample image.

6. The method according to claim 5, characterized in that, The fusion network includes a first fusion network and a residual fusion network; The process of fusing the sample image features and the text semantic features by the fusion network to obtain sample fusion features includes: The first fusion network performs weighted fusion of the sample image features and the text semantic features to obtain reference sample fusion features; The residual fusion network performs residual fusion processing on the reference sample fusion features to obtain the sample fusion features.

7. A method for detecting mold penetration, characterized in that, include: Acquire the image to be detected from the 3D model; The image to be detected is subjected to clipping detection by a clipping detection model to obtain the clipping detection result of the image to be detected. The clipping detection model is trained according to the method described in any one of claims 1 to 6.

8. The method according to claim 7, characterized in that, The step of performing clipping detection on the image to be detected using a clipping detection model and outputting the clipping detection result of the image to be detected includes: The visual encoder in the penetration detection model extracts features from the image to be detected to obtain image features; The text encoder in the mold detection model performs text encoding on the text information corresponding to the image to be detected to obtain text encoding features; the text information corresponding to the image to be detected includes at least one of the image content description text and layout description text of the image to be detected. The fusion output network in the clipping detection model classifies the image based on the image features and the text encoding features to obtain the clipping detection result of the image to be detected.

9. The method according to claim 7, characterized in that, The step of performing clipping detection on the image to be detected using a clipping detection model and outputting the clipping detection result of the image to be detected includes: The visual encoder in the penetration detection model extracts features from the image to be detected to obtain image features; The fusion output network in the model classifies the image based on its features to obtain the detection result of the image to be detected.

10. The method according to any one of claims 7 to 9, characterized in that, The image to be detected is captured by a virtual camera located in a virtual scene; the clipping detection result indicates that there is clipping in the image to be detected, and indicates the corresponding clipping pixel position information; After performing clipping detection on the image to be detected using a clipping detection model and outputting the clipping detection result of the image to be detected, the method further includes: Based on the camera parameters of the virtual camera, the clipping region represented by the clipping pixel position information is transformed to the world coordinate system corresponding to the virtual scene to determine the target mesh surface corresponding to the clipping pixel position information in the virtual scene; Based on the position information of the target mesh surface, generate clipping prompt information.

11. A model training device, characterized in that, include: The first acquisition module is used to acquire multiple sample images and label information corresponding to each sample image. The label information includes sample text information and clipping annotation information corresponding to the sample image. The clipping annotation information is used to indicate whether there is clipping in the 3D model presented in the sample image. The sample text information includes at least one of the sample image content description text and sample layout description text of the sample image. The first feature extraction module is used to extract features from the sample image by the visual encoder in the penetration detection model to obtain sample image features; The first text encoding module is used to encode the sample text information by the text encoder in the penetration detection model to obtain text semantic features; The first clipping detection module is used to perform clipping detection based on the sample image features and the text semantic features by the fusion output network in the clipping detection model, and obtain the clipping detection result of the sample image; The loss determination module is used to calculate the detection loss based on the penetration detection results of the sample image and the corresponding penetration annotation information; An adjustment module is used to adjust the parameters of the penetration detection model according to the detection loss until the training termination condition is met.

12. A device for detecting mold penetration, characterized in that, include: The second acquisition module is used to acquire the image to be detected from the 3D model. The second clipping detection module is used to perform clipping detection on the image to be detected using a clipping detection model to obtain the clipping detection result of the image to be detected. The clipping detection model is trained according to the method described in any one of claims 1 to 6.

13. An electronic device, characterized in that, include: processor; A memory storing computer instructions that, when executed by the processor, implement the method as described in any one of claims 1-10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-10.

15. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method described in any one of claims 1-10.

Images