Object recognition method and device, storage medium and electronic equipment

Abstract

The invention discloses an object recognition method and device, a storage medium and electronic equipment. The object recognition method comprises the following steps: acquiring a first image and a second image obtained by performing image acquisition on a to-be-recognized main body; inputting the first image into a first segmentation model and a second segmentation model, and inputting the second image into the second segmentation model; acquiring first position area information of the position of the target object output by the first segmentation model, and acquiring anchoring information of a first architecture space coordinate system matched with the first image and anchoring information of a second architecture space coordinate system matched with the second image, which are determined based on a segmentation result output by the second segmentation model; and based on the first position area information of the position where the target object is located, the anchoring information of the first architecture space coordinate system and the anchoring information of the second architecture space coordinate system, identifying the repair classification result of the target object, and solving the technical problem of low accuracy of the object identification result in the related technology.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an object recognition method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] With the continuous upgrading of image acquisition technology, the processing precision of image acquisition equipment has been greatly improved. In many application fields, image analysis is often performed on images acquired through the above methods to identify abnormal objects with defects.

[0003] In images containing multiple objects with complex structural relationships, the commonly used image recognition method is a single-focus approach. This involves identifying each object in the image individually and then performing defect detection on specific objects. However, this single-focus method often ignores the structural relationships between objects, leading to positional deviations in the identified abnormal objects and resulting in low accuracy in object recognition.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an object recognition method and apparatus, a storage medium and an electronic device, to at least solve the technical problem of low accuracy of object recognition results in related technologies.

[0006] According to one embodiment of this application, an object recognition method is provided, comprising: acquiring a first image and a second image obtained by image acquisition of a subject to be identified, wherein the subject includes multiple objects with functional relationships, and the second image is an image acquired after the first image; inputting the first image into a first segmentation model and a second segmentation model, and inputting the second image into the second segmentation model, wherein the first segmentation model is obtained by training an initialized segmentation model using a first sample set for segmenting target objects that meet object anomaly conditions from the image, and the second segmentation model is obtained by training an initialized segmentation model using a second sample set for segmenting supporting objects from the image. The structural framework includes a first sample set containing location labels of the target object's location, and a second sample set containing location labels of each anchor structure within the structural framework. The system acquires first location region information of the target object's location output by the first segmentation model, and obtains anchoring information of a first architecture spatial coordinate system matching the first image and a second architecture spatial coordinate system matching the second image, determined based on the segmentation results output by the second segmentation model. Based on the first location region information of the target object's location, the anchoring information of the first architecture spatial coordinate system, and the anchoring information of the second architecture spatial coordinate system, the system identifies the repair classification result of the target object.

[0007] According to another aspect of the present invention, an object recognition device is also provided, comprising: a first acquisition unit for acquiring a first image and a second image obtained by image acquisition of a subject to be recognized, wherein the subject includes a plurality of objects having functional relationships, and the second image is an image acquired after the first image; and a segmentation unit for inputting the first image into a first segmentation model and a second segmentation model, and inputting the second image into the second segmentation model, wherein the first segmentation model is obtained by training an initialized segmentation model using a first sample set for segmenting target objects that satisfy object anomaly conditions from the image, and the second segmentation model is obtained by training an initialized segmentation model using a second sample set for segmenting objects that support objects from the image. The structural framework includes a first sample set containing location labels of the target object's location, and a second sample set containing location labels of each anchor structure within the structural framework. A second acquisition unit acquires the first location region information of the target object output by the first segmentation model, and acquires the anchoring information of the first architectural spatial coordinate system matching the first image, and the anchoring information of the second architectural spatial coordinate system matching the second image, determined based on the segmentation results output by the second segmentation model. A recognition unit identifies the repair classification result of the target object based on the first location region information of the target object's location, the anchoring information of the first architectural spatial coordinate system, and the anchoring information of the second architectural spatial coordinate system.

[0008] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described object recognition method at runtime.

[0009] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the object recognition method described above through the computer program.

[0010] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described object recognition method.

[0011] In this embodiment, a first image and a second image are obtained by image acquisition of the subject to be identified. The first image is input into a first segmentation model and a second segmentation model, and the second image is input into the second segmentation model. Using the specifically trained first and second segmentation models, the target object and structural frame are identified and segmented, enhancing the specificity and accuracy of the identification. The first segmentation model focuses on identifying abnormal objects, while the second segmentation model focuses on stable structural frames. The combined use of the two segmentation models can more accurately locate and evaluate changes in the target object. The first location region information of the target object output by the first segmentation model is obtained, and the segmentation result determined based on the output of the second segmentation model is obtained. The anchoring information of the first architectural spatial coordinate system matched with the first image and the anchoring information of the second architectural spatial coordinate system matched with the second image, through positioning and anchoring, can maintain relative consistency of recognition results even in the face of complex structural and positional changes, thereby improving recognition accuracy. Furthermore, based on the first location region information of the target object's location, the anchoring information of the first architectural spatial coordinate system, and the anchoring information of the second architectural spatial coordinate system, the repair and classification results of the target object can be identified. Through precise coordinate registration and before-and-after comparison, the influence of positional deviation on the recognition results can be avoided, improving the accuracy of the identified repair and classification results, and solving the technical problem of low accuracy of object recognition results in related technologies. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the hardware environment for an optional object recognition method according to an embodiment of this application;

[0015] Figure 2 This is a flowchart of an optional object recognition method according to an embodiment of this application;

[0016] Figure 3 This is a schematic diagram of an optional image segmentation according to an optional embodiment of this application;

[0017] Figure 4 This is a schematic diagram illustrating an optional coordinate system construction according to an optional embodiment of this application;

[0018] Figure 5 This is a schematic diagram illustrating another optional coordinate system construction according to an optional embodiment of this application;

[0019] Figure 6 This is a schematic diagram of an optional classification model according to an optional embodiment of this application;

[0020] Figure 7 This is a schematic diagram of an optional object recognition method according to an embodiment of this application;

[0021] Figure 8 This is a schematic diagram of an optional object recognition device according to an embodiment of the present invention;

[0022] Figure 9 This is a schematic diagram of the components of an optional electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] The collection, storage, use, processing, transmission, provision, and disclosure of user data and other information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0026] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0027] According to one aspect of the embodiments of this application, an object recognition method is provided. Optionally, as an optional implementation, the above-described object recognition method may be applied to, but is not limited to, [examples of other methods]. Figure 1 The object recognition system shown consists of terminal device 102, server 104, and network 110. For example... Figure 1 As shown, terminal device 102 communicates with server 104 via network 110. This network can include, but is not limited to, wired networks and wireless networks. The wired network includes local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). The wireless network includes Bluetooth, Wi-Fi, and other networks enabling wireless communication. The terminal device can include, but is not limited to, at least one of the following: mobile phones (such as Android phones, iOS phones, etc.), laptops, tablets, handheld computers, MIDs (Mobile Internet Devices), PADs, desktop computers, smart TVs, in-vehicle devices, etc. Terminal device 102 can include, but is not limited to, a display, processor, and memory. Server 104 can be a single server, a server cluster consisting of multiple servers, or a cloud server. The server includes a database and a processing engine.

[0028] The specific process can be as follows: Step S102, acquire the first image and the second image obtained by image acquisition of the subject to be identified, wherein the subject includes multiple objects with functional relationships, and the second image is an image acquired after the first image;

[0029] Step S104: Input the first image into the first segmentation model and the second segmentation model, and input the second image into the second segmentation model. The first segmentation model is obtained by training the initialized segmentation model using the first sample set to segment the target object that meets the object anomaly condition from the image. The second segmentation model is obtained by training the initialized segmentation model using the second sample set to segment the structural frame that supports the object from the image. The first sample set includes first samples carrying location labels of the target object's location, and the second sample set includes second samples carrying location labels of each anchoring structure in the structural frame.

[0030] Step S106: Obtain the first location region information of the target object output by the first segmentation model, and obtain the anchoring information of the first architecture spatial coordinate system that matches the first image and the second architecture spatial coordinate system that matches the second image, determined based on the segmentation result output by the second segmentation model.

[0031] Step S108: Based on the first location area information of the target object, the anchoring information of the first architecture space coordinate system and the anchoring information of the second architecture space coordinate system, the repair classification result of the target object is identified.

[0032] In step S110, server 104 executes S110 via network 110, sending the repair classification result of the identified target object to terminal device 102, where the collected image information can be displayed.

[0033] remove Figure 1 In addition to the examples shown, the above steps can be completed independently by the client or the server, or jointly by the client and the server. The terminal device 102 includes, but is not limited to, handheld devices (such as mobile phones), laptops, desktop computers, in-vehicle devices, etc. This application does not limit the specific implementation of the terminal device 102.

[0034] This embodiment provides an object recognition method, which is applied to the aforementioned computer terminal. Figure 2 This is a flowchart of an object recognition method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0035] Step S202: Obtain a first image and a second image obtained by image acquisition of the subject to be identified, wherein the subject includes multiple objects with functional relationships, and the second image is an image acquired after the first image;

[0036] Step S204: Input the first image into the first segmentation model and the second segmentation model, and input the second image into the second segmentation model. The first segmentation model is obtained by training the initialized segmentation model using the first sample set to segment the target object that meets the object anomaly condition from the image. The second segmentation model is obtained by training the initialized segmentation model using the second sample set to segment the structural frame that supports the object from the image. The first sample set includes first samples carrying location labels of the target object's location, and the second sample set includes second samples carrying location labels of each anchoring structure in the structural frame.

[0037] Step S206: Obtain the first location region information of the target object output by the first segmentation model, and obtain the anchoring information of the first architecture spatial coordinate system that matches the first image and the second architecture spatial coordinate system that matches the second image, determined based on the segmentation result output by the second segmentation model.

[0038] Step S208: Based on the first location region information of the target object's location, the anchoring information of the first architecture spatial coordinate system, and the anchoring information of the second architecture spatial coordinate system, the repair classification result of the target object is identified.

[0039] It should be noted that the object processing method of this application can be applied to a variety of scenarios, including the analysis and recognition of pathological images, the detection and recognition of material structural defects, and the recognition of road environment objects, etc., without any specific limitations.

[0040] In step S202 above, the subject can be a pedestrian on the road, a building structure, or an object to be identified and analyzed. The first image and the second image are images of the subject collected at different time points or under different conditions, typically used for before-and-after comparison or time series analysis. For example, two sets of image data, namely the first image and the second image, are collected from the subject to be identified. The first image can be image data under conditions such as the initial state or before treatment, while the second image can be collected at a subsequent time point or after treatment, for comparison and analysis.

[0041] In step S204 above, the first segmentation model is used to segment the target object model from the image that meets the specific abnormal conditions of the object, such as abnormal structures in buildings, abnormally moving vehicles on roads, or lesions in the human body; the second segmentation model is used to segment the structural frame or reference model supporting the target object from the image, such as fixed signs in road scenes, the femoral head or spine in the human body. As an optional implementation, a pre-trained first segmentation model can be used to process the first image to identify and segment the target object (such as a tumor), and the second segmentation model can be used to process the first and second images to identify and segment the structural frame or reference model (such as the femoral head). Figure 3 The automatic tumor segmentation model shown segments the pre-treatment T2W image, obtaining the crescent-shaped marker within the white box in the first image on the right, corresponding to the tumor. The automatic femoral head segmentation model segments both the pre-treatment and post-treatment T2W images, obtaining their respective segmentation results, as shown below. Figure 3 The second and third images on the right, marked with a semi-circular white box, represent the femoral head segmentation results.

[0042] Understandably, in the field of materials science, it is necessary to analyze the distribution of defects in materials under different conditions, such as cracks in metals or pores in composite materials. The first segmentation model mentioned above can be used to identify defect regions in materials, such as cracks. The first segmentation model is trained based on a first sample set, which marks the specific locations of defects. The second segmentation model mentioned above is used to identify stable structures in materials, such as material edges or specific marker points, for subsequent comparative analysis. The second segmentation model is trained based on a second sample set, which includes images of stable component structures.

[0043] In step S206 above, the first location region information is, for example, the coordinate information of the center point and edge of the tumor; the first spatial coordinate system matched with the first image is, for example, the imaging reference system x1-y1-z1 before treatment, and the anchoring information includes, but is not limited to, the coordinates of the center point of the femoral head, the coordinates of the center of the femoral head section, the tumor location coordinates, and the direction of the coordinate axes in the first spatial coordinate system; the second spatial coordinate system matched with the second image is, for example, the imaging reference system x2-y2-z2 after treatment, and the anchoring information includes, but is not limited to, the coordinates of the center point of the femoral head, the coordinates of the center of the femoral head section, and the direction of the coordinate axes in the second spatial coordinate system.

[0044] In step S208 above, spatial registration and analysis are performed using the positional information of the target object (such as a tumor) in the first image and the anchoring information of the structural frame (such as the femoral head) in the first and second structural coordinate systems to identify the repair classification result of the target object. The classification result includes, but is not limited to, complete pathological remission, partial pathological remission, and no pathological remission in the medical field, and includes, but is not limited to, structural safety, minor damage, and severe damage in the field of material structure.

[0045] As an optional implementation, in medical scenarios, the positional information based on the femoral head is used to transform the tumor region's position before treatment to a post-treatment coordinate system to assess changes in the tumor region. For example, the vertex coordinates of the tumor bounding box are transformed from the (x1-y1-z1) coordinate system to the (x'-y'-z') coordinate system constructed based on femoral head segmentation using a coordinate transformation formula. Specifically, based on the positional relationships such as rotation and translation between different coordinate systems, the position of the tumor in the (x2-y2-z2) coordinate system after treatment is calculated, thereby obtaining image information of this position. Based on the pre- and post-treatment image information, the lesion repair classification results can be identified.

[0046] The above steps S202-S208 are fully described using an optional implementation method, for example, to identify the repair results of a bridge: Before the repair work begins, a high-precision UAV carrying multimodal sensors (such as RGB cameras, infrared cameras, etc.) is used to acquire detailed images of the bridge, obtaining a first image that records the original state of the bridge structure, including abnormalities such as cracks and corrosion. After the repair is completed, the same equipment is used again to acquire a second image at the same angle and viewpoint to record the structural state after the repair.

[0047] The first segmentation model, trained using images of the bridge in various damage states (first sample set), can accurately identify and segment abnormal objects in the bridge, such as cracked areas and corroded parts. The second segmentation model, trained using images of the bridge's supporting structures (second sample set), can be used to identify these supporting structures, such as piers and bridge deck beams. The first image is input into both the first and second segmentation models, while the second image is only input into the second segmentation model. The first segmentation model outputs the locations of abnormal objects on the bridge before repair, while the second segmentation model outputs the locations of the bridge's supporting structures before and after repair.

[0048] Based on the specific location areas of abnormal objects such as cracks and corrosion output by the first segmentation model, and the anchoring information of key structures such as piers or beams determined by the second segmentation model, the position and range of abnormal objects in the first structural space coordinate system before repair can be determined. Then, by calculating the coordinate transformation matrix, the position and range of abnormal objects before repair are transformed to the second structural space coordinate system after repair. Specifically, the coordinate positions of abnormal structures in the first structural space coordinate system matched with the first image are first transformed to a reference coordinate system constructed based on key structures such as piers or beams, and then transformed from the reference coordinate system to the second structural space coordinate system matched with the second image to obtain the position of the abnormal structure after repair. Then, the condition of the same fault repair area in the first and second images is compared. Finally, the changes of abnormal objects before and after repair are analyzed to determine the repair classification result, such as "completely repaired", "partially repaired", or "not repaired".

[0049] In the identification process described above, a specialized segmentation model is trained to identify and locate abnormal objects (such as cracks and damaged areas) and supporting structural frames in the building structure. The models are segmented in the images before and after repair. The coordinate system constructed based on the structural frame provides a stable reference for the location of abnormal objects. Even when the bridge is slightly deformed or the viewing angle changes, the accuracy of identification and registration can be maintained, avoiding the identification bias that may exist in a single model. By combining the image analysis before and after repair, the model provides accurate repair classification results based on the changes in abnormal objects and the state of the structural frame.

[0050] In this embodiment, a first image and a second image are obtained by image acquisition of the subject to be identified. The first image is input into a first segmentation model and a second segmentation model, and the second image is input into the second segmentation model. Using the specifically trained first and second segmentation models, the target object and structural frame are identified and segmented, enhancing the specificity and accuracy of the identification. The first segmentation model focuses on identifying abnormal objects, while the second segmentation model focuses on stable structural frames. The combined use of the two segmentation models can more accurately locate and evaluate changes in the target object. The first location region information of the target object output by the first segmentation model is obtained, and the segmentation result determined based on the output of the second segmentation model is obtained. The anchoring information of the first architectural spatial coordinate system matched with the first image and the anchoring information of the second architectural spatial coordinate system matched with the second image, through positioning and anchoring, can maintain relative consistency of recognition results even in the face of complex structural and positional changes, thereby improving recognition accuracy. Furthermore, based on the first location region information of the target object's location, the anchoring information of the first architectural spatial coordinate system, and the anchoring information of the second architectural spatial coordinate system, the repair and classification results of the target object can be identified. Through precise coordinate registration and before-and-after comparison, the influence of positional deviation on the recognition results can be avoided, improving the accuracy of the identified repair and classification results, and solving the technical problem of low accuracy of object recognition results in related technologies.

[0051] As an optional approach, based on the first location region information of the target object's location, the anchoring information of the first architectural spatial coordinate system, and the anchoring information of the second architectural spatial coordinate system, the repair classification results of the target object are identified as follows:

[0052] S1, obtain the first coordinate transformation matrix between the first architecture space coordinate system and the reference architecture space coordinate system, and the second coordinate transformation matrix between the reference architecture space coordinate system and the second architecture space coordinate system, wherein the reference architecture space coordinate system is determined based on the key anchoring structure in the main body's structural framework;

[0053] S2, transform the first position region information of the target object according to the first coordinate transformation matrix to obtain the reference position region coordinates of the target object in the reference architecture space coordinate system;

[0054] S3, transform the coordinates of the reference position area according to the second coordinate transformation matrix to obtain the second position area information of the projection position of the target object in the second image;

[0055] S4, input the first image region indicated by the first position region information in the first image and the second image region indicated by the second position region information in the second image into the object classification model, wherein the object classification model is used to determine the repair classification result of the target object based on the comparison result between the first image region and the second image region.

[0056] Optionally, in step S1 above, a reference frame space coordinate system is constructed based on the stable anchoring structure of the femoral head. This means that the position of the femoral head structure remains unchanged before and after treatment. Using a three-dimensional registration algorithm, two transformation matrices are calculated. The first coordinate transformation matrix is ​​used to transform the positional information (such as tumor location) in the first frame space coordinate system to the reference frame space coordinate system, while the second coordinate transformation matrix is ​​used to transform the positional information in the reference frame space coordinate system to the second frame space coordinate system.

[0057] In steps S1-S3 above, for example, when it is necessary to compare the minute changes of the material before and after cyclic stress, two different coordinate systems are used to record the key point positions on the material surface. First framework space coordinate system: the coordinate system of the material before cyclic stress (A), second framework space coordinate system: the coordinate system of the material after cyclic stress (B), reference framework space coordinate system: a unified coordinate system constructed based on stable feature points on the material surface (C).

[0058] Establish a reference architecture spatial coordinate system (C): By selecting a set of stable feature points on the material surface (such as engravings or special markings on the material surface), marking them before and after the action, a unified reference architecture spatial coordinate system (C) is established based on these feature points.

[0059] Calculate the coordinate transformation matrix: Use the 3D registration algorithm to calculate the first coordinate transformation matrix (M1) from coordinate system (A) to (C) and the second coordinate transformation matrix (M2) from coordinate system (C) to (B).

[0060] Coordinate Transformation: Assume the position coordinates of a key point on the material surface in coordinate system (A) are (10, 20, 30). First, transform (10, 20, 30) to coordinate system (C) using the first coordinate transformation matrix (M1), and let the transformed coordinates be (15, 25, 35). Then, use the second coordinate transformation matrix (M2) to transform (15, 25, 35) to coordinate system (B), obtaining the actual position coordinates of the key point after applying cyclic stress (12, 22, 32). No specific restrictions are placed on the above coordinate transformation process.

[0061] In step S4 above, for example, after obtaining the location information of the target object before treatment (first image region) and after treatment (second image region), the image data of these two regions are input into the object classification model. The object classification model is, for example, a deep learning model, such as a convolutional neural network (CNN), which automatically extracts and compares the image features of the two regions, such as changes in texture, shape, and size, to determine the repair classification result of the target object. For example, whether the tumor has shrunk, whether the shape has changed, whether pathological complete remission has been achieved, whether new metastatic lesions have appeared, etc. The image data of the overlapping region is analyzed by a three-dimensional convolutional neural network to identify the target object repair classification probability result based on image features.

[0062] By determining the transformation relationship between different spatial coordinate systems, a unified coordinate reference system is established, providing a foundation for subsequent accurate comparison and spatial positioning. Even in images acquired at different times, the consistency of the anchoring structure can be maintained, avoiding object recognition deviations caused by coordinate system differences. Through two coordinate transformations, the positional information of the target object is accurately matched in different images, effectively avoiding recognition errors caused by positional differences. Using the accurately matched image regions, the object classification model can accurately assess the degree of restoration of the target object, providing scientific classification results and solving the problem of inaccurate recognition.

[0063] As an optional approach, after inputting the first image into the first segmentation model and the second segmentation model, and inputting the second image into the second segmentation model, the method further includes:

[0064] S1, Determine the anchoring information of the first architecture spatial coordinate system by the position of each anchoring structure in the first image output by the second segmentation model;

[0065] S2, Based on the first anchoring position of the key anchoring structure in the segmentation result of the first image, construct the first reference architecture spatial coordinate system;

[0066] S3, calculate the first coordinate transformation matrix from the first architecture space coordinate system to the first reference architecture space coordinate system;

[0067] S4. Determine the anchoring information of the second architecture spatial coordinate system from the positions of each anchoring structure in the second image output by the second segmentation model.

[0068] S5, Based on the second anchoring position of the key anchoring structure in the segmentation result of the second image, construct the second reference architecture spatial coordinate system;

[0069] S6, calculate the second coordinate transformation matrix from the second reference architecture space coordinate system to the first architecture space coordinate system;

[0070] The first reference architecture space coordinate system and the second reference architecture space coordinate system are the same coordinate system, and both are reference architecture space coordinate systems.

[0071] In step S1 above, the anchoring information of the first architecture spatial coordinate system is the coordinate origin, x1 axis, y1 axis, z1 axis, etc. in the default coordinate system constructed based on the fixed rules set for medical equipment for pre-treatment images.

[0072] In step S2 above, the first anchoring position of the key anchoring structure in the first image, such as the femoral head, is determined by using the position coordinate information of the femoral head, including the coordinate position of the center of the largest cross-section of both femoral heads, and then the origin, x' axis, y' axis, z' axis, etc. of the first reference architecture spatial coordinate system are determined, and the first reference architecture spatial coordinate system is constructed.

[0073] In step S3 above, a three-dimensional registration algorithm (such as the Iterative Closest Point (ICP) algorithm, rigid body registration, or elastic body registration) is used to calculate the mathematical transformation from the first architecture spatial coordinate system to the first reference architecture spatial coordinate system based on the position and orientation of the anchoring structure in the two coordinate systems. This is the first coordinate transformation matrix. Specifically, it includes the calculation of the translation vector: finding the spatial distance between the midpoints of the anchoring structure in the two coordinate systems to determine the translation amount required for the transformation; and the calculation of the rotation matrix: determining the required rotation angle and axis by calculating the angle between the principal axes of the anchoring structure in the two coordinate systems, thereby deriving the rotation matrix. No specific restrictions are imposed here.

[0074] In step S4 above, the anchoring information of the second architecture spatial coordinate system is the post-treatment imaging device (such as MRI, CT machine), and the coordinate origin, x2 axis, y2 axis, z2 axis, etc. in the default coordinate system constructed based on the fixed rules set for medical devices for post-treatment images.

[0075] In step S5 above, the second anchoring position of the key anchoring structure in the second image, such as the femoral head, is determined by the coordinate information of the femoral head, including the coordinate position of the center of the largest cross-section of both femoral heads. The origin, x' axis, y' axis, z' axis, etc. of the second reference frame spatial coordinate system are then determined, and the second reference frame spatial coordinate system is constructed. It can be understood that the position coordinate information of the femoral head remains basically unchanged, that is, the first anchoring position and the second anchoring position are equivalent.

[0076] In step S6 above, the second coordinate transformation matrix for transforming from the second reference architecture space coordinate system to the first architecture space coordinate system is determined based on the translation vector calculation and rotation matrix calculation in step S3.

[0077] The above process is described below using a specific implementation method:

[0078] Establishment of an individual reference frame: A reference frame for the patient's own femur is constructed using automatically segmented bilateral femoral head ROIs. The relative motion displacement of the patient between imaging sessions at different times does not affect the coordinate positioning within this reference frame. The femoral reference frame is defined as follows: (a) Find the largest cross-section of the bilateral femoral head ROIs, i.e. Figure 4 (a) Calculate the center position of the circle from the inner (semi)circular edge of the largest cross section; (b) Define the straight line connecting the two centers as the x' axis, the right direction as the positive direction, and the midpoint of the two centers as the origin; (c) Calculate the perpendicular plane of the x' axis, the intersection of this plane and the imaging reference plane xz is defined as the z' axis, and the head direction is defined as the positive direction; (d) Define the perpendicular direction of x'-y'-z' as the y' direction, the front of the supine position as the positive direction, and the x'-y'-z' direction satisfies the right-hand rule, with coordinates in millimeters. Figure 4 The procedure for establishing a patient's own femoral reference frame based on pre-neoadjuvant T2W images is described, where x1-y1-z1 is the pre-neoadjuvant imaging reference frame.

[0079] Based on an individual reference system, the pre-treatment tumor region can be used to locate the post-treatment tumor region: Similarly, the patient's own femoral reference system can also be established from T2W images after neoadjuvant therapy (e.g., Figure 5 The process is shown below, where x2-y2-z2 is the imaging reference frame after neoadjuvant therapy. Based on the assumption that the femoral reference frame remains unchanged before and after treatment, the tumor region after neoadjuvant therapy can be located in the post-treatment T2W image. The specific method is as follows: (a) Generate a three-dimensional rectangular bounding box for the automatically segmented tumor ROI within the pre-neoadjuvant therapy imaging reference frame x1-y1-z1, transform it to the femoral reference frame x'-y'-z', and calculate the positions of the 8 vertices of the bounding box in x'-y'-z'. (b) Calculate the transformation rule of the femoral x'-y'-z' reference frame within the post-neoadjuvant therapy imaging reference frame x2-y2-z2, and calculate its coordinates in x2-y2-z2 based on the positions of the 8 vertices in x'-y'-z', thereby obtaining the position of the tumor in the post-treatment coordinate system.

[0080] It should be noted that, Figure 4 , Figure 5 In the diagram, Axial View is the axial view / cross-sectional view, Coronal View is the coronal view / frontal view, and Sagittal View is the sagittal view / midline view.

[0081] Determining the anchoring information involves locating key anchoring structures in images before and after treatment, providing reference points for coordinate transformation, and ensuring the stability of the recognition process. Establishing a reference coordinate system defines a unified coordinate system as a common framework for image registration and target object recognition before and after treatment, reducing errors caused by coordinate inconsistencies. By calculating the transformation matrix, accurate registration of images in spatial coordinates is achieved, maintaining high accuracy in object recognition even in the face of complex changes in body position or adjustments to equipment parameters.

[0082] As an optional approach, S1, based on the first anchoring position of the key anchoring structure in the segmentation result of the first image, constructing the first reference architecture spatial coordinate system includes: determining the first position of the first part of the key anchoring structure in the first image and the second position of the second part of the key anchoring structure; determining the center of the line connecting the first position and the second position as the center of the first reference architecture spatial coordinate system, and constructing the three-dimensional direction vector of the first reference architecture spatial coordinate system based on the tangent of the line at the center;

[0083] S2, based on the second anchoring position of the key anchoring structure in the segmentation result of the second image, constructing the second reference architecture spatial coordinate system includes: determining the first position of the first part of the key anchoring structure in the second image and the second position of the second part of the key anchoring structure; determining the center of the line connecting the first position and the second position as the center of the second reference architecture spatial coordinate system, and constructing the three-dimensional direction vector of the second reference architecture spatial coordinate system based on the tangent of the line at the center.

[0084] In step S1 above, the first location of the first part of the key anchoring structure in the first image and the second location of the second part of the key anchoring structure are determined; the center of the line connecting the first and second locations is determined as the center of the first reference architecture spatial coordinate system, and a three-dimensional direction vector of the first reference architecture spatial coordinate system is constructed based on the cross-section of the connecting line at the center. Optionally, the key anchoring structure mentioned above can be a relatively stable and minimally changing structure in medical imaging, such as the femoral head or spine; in the field of material structure, it can be a component with strong structural stability that is not easily changed, without specific limitations. The first and second parts mentioned above can be the same location, such as the center of the cross-section of both femoral heads, or the upper or lower end of the femoral head, etc.

[0085] As an optional implementation, in the construction field, when selecting a structure, two stable and non-deformable components identified from the pre-construction image (first image) can be selected as reference points, such as two bridge piers. The selected first and second parts correspond to specific parts of the two piers, for example, the center point of the widest cross-section at the bottom of each pier in contact with the ground. The midpoint of the line connecting the first and second parts is calculated, and this midpoint will be used as the center position of the first reference structure spatial coordinate system. A three-dimensional direction vector is constructed: the direction of the line connecting the piers is defined as the x-axis, with the positive direction from left to right. Since the bridge is parallel to the ground, the z-axis is defined as perpendicular to the ground, with the positive direction upward. According to the right-hand rule, the y-axis is perpendicular to the plane formed by the x-axis and z-axis, and the positive direction can be set to the direction of the water surface towards the side of the bridge deck.

[0086] In step S2 above, the first position of the first part of the key anchoring structure and the second position of the second part of the key anchoring structure in the second image are determined; the center of the line connecting the first and second positions is determined as the center of the second reference structure spatial coordinate system, and the three-dimensional direction vector of the second reference structure spatial coordinate system is constructed based on the tangent plane of the connecting line at the center. The same method as constructing the coordinate system in S1 above can be used to select two stable and deformation-resistant bridge piers identified by segmentation in the post-construction image (second image), and the center point of the widest cross-section of the bottom of each pier in contact with the ground, thereby constructing the three-dimensional direction vector of the second reference structure spatial coordinate system.

[0087] By establishing a unified coordinate system based on the positions of the first and second parts of the key anchoring structure as the spatial positioning frame in the first image, and using the same method in the second image, constructing a coordinate system consistent with the spatial coordinate system of the first reference architecture based on the same anchoring structure parts as the positional reference for the post-treatment image, and using the center of the line connecting the anchoring structure parts as the center, the spatial coordinate systems of the first and second reference architectures are constructed. This ensures that the images before and after have a unified three-dimensional spatial positioning standard. The two coordinate systems are not only consistent in definition, but also have unified specifications in spatial positioning and direction. This effectively overcomes the problem of inaccurate recognition caused by factors such as equipment differences and position changes.

[0088] As an optional approach, the first image region indicated by the first location region information in the first image and the second image region indicated by the second location region information in the second image are input into the object classification model, including:

[0089] S1, In the object classification model, the first image region and the second image region are compared to obtain the repair parameters of the target object, wherein the repair parameters are used to indicate the degree of repair of the target object;

[0090] S2, determine the repair classification result of the target object based on the repair parameters of the target object, wherein the repair classification result includes complete repair and incomplete repair.

[0091] In step S1 above, the object classification model can be a trained machine learning model, such as a convolutional neural network (CNN), used to identify and classify target objects in medical images, engineering structure images, etc., and to evaluate their state changes. The first image region and the second image region refer to the image data of the target object's location before treatment or repair (first image) and after treatment or repair (second image), respectively. The target object refers to a specific structure or lesion for which the degree of repair needs to be evaluated, such as a crack in a bridge or a tumor boundary. The repair parameters are quantitative indicators obtained through model comparative analysis, used to indicate the degree of repair of the target object, and can be various indicators, such as size change, strength recovery value, integrity score, etc.

[0092] As an optional implementation, in step S2 above, it is assumed that the repair parameter output by the model is that the crack width is reduced by 80%, and the preset "complete repair" standard is that the crack width is reduced by more than 90%. Since the crack width reduction of 80% does not meet the standard of "complete repair", the repair classification result is "incomplete repair".

[0093] This application resamples the images within the bounding boxes of all modalities (T2W, DWI, ADC) before and after treatment into a 128×128×16 three-dimensional matrix. Specifically, all images before and after treatment are combined into a 6×16×128×128 four-dimensional matrix as input data.

[0094] The structure of a convolutional neural network is as follows: Figure 6As shown, the model includes: Conv3D (3D Convolutional Layer): Applying convolutional kernels in three dimensions (width, height, time, or depth) to extract features from volumetric data. When processing medical images, Conv3D layers can simultaneously consider the X, Y, and Z axis information of the image, capturing the features of the 3D structure. BN (Batch Normalization): Used to standardize input data. Batch normalization standardizes the mean and variance of each batch of data, keeping the input of each layer of the network relatively stable, thereby stabilizing and accelerating the training process. MP3D (3D Max Pooling Layer): MP3D is a pooling layer in convolutional neural networks. It extracts the maximum value from 3D data to reduce the size of the feature map while retaining the most important features. CC (Channel Concatenation): In neural networks, CC layers are used to concatenate multiple feature maps along the channel dimension to generate a feature map containing more information. Feature Extracting Unit: Refers to the component or structure in the network used to extract data features. Feature extraction units may consist of a combination of multiple Conv3D, BN, and MP3D layers, used to extract high-level abstract features from the input data. Flattening layers convert 3D or multi-dimensional feature maps into 1D vectors so they can be input into fully connected layers. Dense fully connected layers convert the flattened feature vectors into classification or regression outputs. Dropout layers randomly "drop" certain nodes in the network during training, reducing inter-neuron dependencies and preventing overfitting.

[0095] By automatically comparing models and quantifying parameters, the degree of repair of target objects can be accurately identified, which not only improves the accuracy of identification but also reduces the interference of human factors, providing a more scientific and objective basis for medical decision-making and engineering evaluation.

[0096] As an optional approach, before acquiring the first and second images obtained from image acquisition of the subject to be identified, the following steps are also included:

[0097] S1, Obtain the third sample set, wherein the third sample set includes the first sample subset and the second sample subset. The first sample subset includes multimodal sample images containing abnormal objects within the subject, and the second sample subset includes multimodal sample images collected after repairing the abnormal objects in the first sample subset.

[0098] S2, use the third sample set to train the initialized classification model until an object classification model that satisfies the third convergence condition is obtained.

[0099] In step S1 above, the third sample set includes a comprehensive sample set of image data before or after treatment or repair, used to train the classification model. The first sample subset is a set of sample images acquired using multimodal imaging techniques when an abnormal object (such as a tumor, injury, or fissure) exists. The second sample subset includes a set of multimodal sample images acquired for the same abnormal object after treatment or repair. The multimodal sample images are image data acquired using multiple imaging techniques (such as MRI, CT, X-ray, and ultrasound) at the same time point or state to comprehensively capture the characteristics and changes of the abnormal object.

[0100] As an optional implementation, in bridge structure monitoring projects, multimodal sample images (such as infrared thermograms, visible light photographs, and structural radar scans) containing cracks are collected from several bridges to form a first sample subset. Then, multimodal sample images after crack repair are collected to form a second sample subset. Each image set ensures that it includes both pre- and post-repair states, as well as detailed classification labels such as crack width, length, and location. The first and second sample subsets are then integrated into a third sample set to ensure the diversity and representativeness of the sample set, covering different types of anomalies and repair situations.

[0101] Optionally, in step S2 above, a CNN model in a deep learning framework such as TensorFlow or PyTorch is selected for initialization, and the input / output layers, hidden layer structure, and initial parameters of the model are set. The CNN model is trained using a third sample set containing samples before and after repair. During training, the model learns how to extract features from multimodal images and identify repair classification results based on these features, such as the degree of crack repair. After each round of training, the model's performance is evaluated on the validation set to check if the model meets the third convergence condition, including setting a performance threshold, such as 90% classification accuracy, or a loss function threshold, such as a cross-entropy loss of 0.01. If the model's performance does not meet the set convergence condition, it may be necessary to adjust the learning rate, increase the number of iterations, or optimize the network structure, and then restart the training process until the model can stably output high-accuracy recognition results on the validation set.

[0102] The establishment of the third sample set provides the model with a rich source of information, enabling it to understand and distinguish between normal and abnormal objects from a multimodal perspective, thereby improving the accuracy and generalization ability of recognition. Then, the prepared third sample set is used to train the initialized classification model. Through iterative learning, the model gradually optimizes its internal parameters, learns to automatically extract features from multimodal images, and recognizes and classifies the degree of repair of target objects. This allows the model to make more accurate and reliable judgments when facing complex and ever-changing recognition scenarios, reducing recognition errors.

[0103] As an optional approach, before inputting the first image into the first segmentation model and the second segmentation model, and before inputting the second image into the second segmentation model, the following steps are also included:

[0104] S1, obtain the first sample set and the second sample set;

[0105] S2, use the first sample set to train the initialized segmentation model until a first segmentation model that satisfies the first convergence condition is obtained;

[0106] S3, use the second sample set to train the initialized segmentation model until a second segmentation model that satisfies the second convergence condition is obtained.

[0107] The above steps S1-S3 will be described using an optional implementation method.

[0108] Pre-treatment MRI images were collected, with the tumor area marked by a specialist, forming the first sample set. Simultaneously, femoral MRI images were collected, with the femoral head region also marked, forming the second sample set.

[0109] Choose a suitable CNN model for the segmentation task, such as U-Net, and randomly initialize its parameters. Model training: Iteratively train the model using data from the first sample set. After each training round, evaluate the model's performance on the validation set, such as calculating the segmentation accuracy. Check convergence: Set a first convergence criterion, for example, if the DSC on the validation set does not improve for 5 consecutive rounds, or the improvement is less than 0.01, the model training is considered converged. Adjust and repeat training: If the model training does not meet the first convergence criterion, adjust the learning rate, increase the number of training rounds, or optimize the network structure, and then repeat the training process until the criterion is met.

[0110] The same model structure can be used, but the parameters are re-initialized randomly to prepare it for the femoral head segmentation task. Model training: Train the model using data from the second sample set, optimizing the model parameters to enable accurate segmentation of the femoral head region. Check the second convergence condition: Set a second convergence condition, such as ensuring the average accuracy of femoral head segmentation stabilizes at a certain level, guaranteeing the model has effectively learned the features of the femoral head. Adjustment and repeated training: If the model training does not meet the second convergence condition, it may be necessary to adjust the learning rate, increase the amount of training data, or optimize the model structure until the model's performance on the validation set meets the preset convergence criteria.

[0111] It can be understood that in other fields, such as architecture, the first segmentation model can be used to segment components with cracks, and the second segmentation model can be used to segment structurally stable components; no specific restrictions are made here.

[0112] Through the above implementation methods, a model specifically designed to identify object features under different conditions or states is trained, improving the accuracy of segmenting each structure, improving the accuracy of determining the spatial location of each structure, and further improving the accuracy of determining the location of the target object in the first and second images under different coordinate system transformations, thus improving the accuracy of the final recognition result.

[0113] The present application will now be described using an optional implementation method, such as... Figure 7 As shown:

[0114] The study is divided into two stages: pre-treatment and post-treatment. In the pre-treatment stage, T2W and DWI images are obtained based on pre-treatment MRI (Magnetic Resonance Imaging). Based on T2W, pre-treatment automatic segmentation of the femoral head and pre-treatment automatic segmentation of the tumor can be performed. Furthermore, the pre-treatment tumor coordinates are obtained based on the pre-treatment spatial reference system and the pre-treatment automatic tumor segmentation results. Based on the analysis of the pre-treatment automatic tumor segmentation results and DWI images, pre-treatment T2W, DWI, and ADC images of the tumor region are obtained.

[0115] For the post-treatment phase, T2W and DWI images are obtained based on post-treatment MRI (Magnetic Resonance Imaging). Automatic femoral head segmentation can also be performed based on T2W. Furthermore, the post-treatment tumor region is located within a post-treatment spatial reference system based on the tumor coordinates obtained before treatment. Specifically, this is achieved through a femoral reference system x'-y'-z' constructed based on the femoral head, undergoing two coordinate transformations. Based on the post-treatment tumor region localization results, T2W, DWI, and ADC images of the post-treatment tumor region are obtained.

[0116] Classification models (such as pCR prediction models) are based on input pre-treatment and post-treatment tumor region T2W, DWI, and ADC images to obtain classification results, such as pCR and non-pCR.

[0117] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0118] According to another aspect of the embodiments of this application, an object recognition apparatus for implementing the above-described object recognition method is also provided. For example... Figure 8 As shown, the device includes:

[0119] The first acquisition unit 802 acquires a first image and a second image obtained by image acquisition of the subject to be identified, wherein the subject includes multiple objects with functional relationships, and the second image is an image acquired after the first image;

[0120] The segmentation unit 804 inputs a first image into a first segmentation model and a second segmentation model, and inputs a second image into the second segmentation model. The first segmentation model is obtained by training an initialized segmentation model using a first sample set to segment target objects that meet the object anomaly conditions from the image. The second segmentation model is obtained by training an initialized segmentation model using a second sample set to segment structural frames that support the object from the image. The first sample set includes first samples carrying location labels of the target object's location, and the second sample set includes second samples carrying location labels of each anchor structure in the structural frame.

[0121] The second acquisition unit 806 acquires the first location region information of the target object output by the first segmentation model, and acquires the anchoring information of the first architecture spatial coordinate system that matches the first image and the second architecture spatial coordinate system that matches the second image, determined based on the segmentation result output by the second segmentation model.

[0122] The identification unit 808 identifies the repair classification result of the target object based on the first location area information of the target object's location, the anchoring information of the first architectural spatial coordinate system, and the anchoring information of the second architectural spatial coordinate system.

[0123] For specific implementation examples, please refer to the examples shown in the object recognition method above. These examples will not be repeated here.

[0124] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described object recognition method is also provided, such as... Figure 9 As shown, the electronic device includes a memory 902 and a processor 904. The memory 902 stores a computer program, and the processor 904 is configured to execute the steps of any of the above method embodiments via the computer program.

[0125] Optionally, in this embodiment, the electronic device may be located in at least one of a plurality of network devices in a computer network.

[0126] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0127] S1, acquire the first image and the second image obtained by image acquisition of the subject to be identified, wherein the subject includes multiple objects with functional relationships, and the second image is an image acquired after the first image;

[0128] S2, input the first image into the first segmentation model and the second segmentation model, and input the second image into the second segmentation model. The first segmentation model is obtained by training the initialized segmentation model using the first sample set to segment the target object that meets the object anomaly condition from the image. The second segmentation model is obtained by training the initialized segmentation model using the second sample set to segment the structural framework that supports the object from the image. The first sample set includes first samples carrying the location labels of the target object's location, and the second sample set includes second samples carrying the location labels of each anchor structure in the structural framework.

[0129] S3, obtain the first location region information of the target object output by the first segmentation model, and obtain the anchoring information of the first architecture spatial coordinate system that matches the first image and the second architecture spatial coordinate system that matches the second image, determined based on the segmentation result output by the second segmentation model.

[0130] S4. Based on the first location region information of the target object, the anchoring information of the first architecture spatial coordinate system and the anchoring information of the second architecture spatial coordinate system, the repair classification result of the target object is identified.

[0131] Alternatively, as those skilled in the art will understand, Figure 9 The components shown are for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 9 This does not limit the components of the aforementioned electronic device. For example, the electronic device may also include components that are more... Figure 9 The more or fewer components shown (such as network interfaces, etc.), or having the same Figure 9 The different configurations shown.

[0132] The memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the object recognition method and apparatus in this embodiment. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, thereby realizing the object recognition method described above. The memory 902 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include memory remotely located relative to the processor 904, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 902 may be used, but is not limited to, for information such as target determination requests and target transmission requests. As an example, such as... Figure 9 As shown, the memory 902 may include, but is not limited to, the first acquisition unit 802, the segmentation unit 804, the second acquisition unit 809, and the identification unit 808 in the object recognition device. Furthermore, it may include, but is not limited to, other module units in the object recognition device, which will not be elaborated upon in this example.

[0133] Optionally, the transmission device 906 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 906 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 906 is a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0134] In addition, the above-mentioned electronic device also includes: a display 908 for displaying the above-mentioned classification information; and a connection bus 910 for connecting the various module components in the above-mentioned electronic device.

[0135] According to one aspect of this application, a computer program product is provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions provided in embodiments of this application.

[0136] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0137] It should be noted that the computer system of the electronic device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0138] A computer system includes a Central Processing Unit (CPU), which performs various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) or loaded from RAM. ROM also stores various programs and data required for system operation. The CPU, ROM, and RAM are interconnected via a bus. Input / output interfaces (I / O interfaces) are also connected to the bus.

[0139] The following components are connected to the input / output interface: input sections including keyboards, mice, etc.; output sections including cathode ray tubes (CRTs), liquid crystal displays (LCDs), and speakers; storage sections including hard drives; and communication sections including network interface cards such as LAN cards and modems. The communication section performs communication processing via a network such as the Internet. Drives are also connected to the input / output interface as needed. Removable media, such as disks, optical discs, magneto-optical discs, semiconductor memories, etc., are installed on the drive as needed so that computer programs read from them can be installed into the storage section as required.

[0140] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions defined in the system of this application.

[0141] Optionally, the aforementioned identification unit 808 includes a third acquisition unit, used to acquire a first coordinate transformation matrix between the first architectural spatial coordinate system and the reference architectural spatial coordinate system, and a second coordinate transformation matrix between the reference architectural spatial coordinate system and the second architectural spatial coordinate system, wherein the reference architectural spatial coordinate system is determined based on the key anchoring structures in the structural framework of the subject; the first position region information of the target object is transformed according to the first coordinate transformation matrix to obtain the reference position region coordinates of the target object in the reference architectural spatial coordinate system; the reference position region coordinates are transformed according to the second coordinate transformation matrix to obtain the second position region information of the projection position of the target object in the second image; the first image region indicated by the first position region information in the first image and the second image region indicated by the second position region information in the second image are input into the object classification model, wherein the object classification model is used to determine the repair classification result of the target object based on the comparison result between the first image region and the second image region.

[0142] Optionally, the third acquisition unit includes a first determination unit, used to determine the anchoring information of the first architecture spatial coordinate system based on the positions of each anchoring structure in the first image output by the second segmentation model; to construct a first reference architecture spatial coordinate system based on the first anchoring positions of key anchoring structures in the segmentation results of the first image; to calculate a first coordinate transformation matrix from the first architecture spatial coordinate system to the first reference architecture spatial coordinate system; to determine the anchoring information of the second architecture spatial coordinate system based on the positions of each anchoring structure in the second image output by the second segmentation model; to construct a second reference architecture spatial coordinate system based on the second anchoring positions of key anchoring structures in the segmentation results of the second image; and to calculate a second coordinate transformation matrix from the second reference architecture spatial coordinate system to the first architecture spatial coordinate system; wherein the first reference architecture spatial coordinate system and the second reference architecture spatial coordinate system are the same coordinate system and are both reference architecture spatial coordinate systems.

[0143] Optionally, the first determining unit includes a coordinate system construction unit, used to determine the first position of the first part of the key anchoring structure in the first image and the second position of the second part of the key anchoring structure; to determine the center of the line connecting the first position and the second position as the center of the first reference architecture spatial coordinate system, and to construct a three-dimensional direction vector of the first reference architecture spatial coordinate system based on the tangent plane of the line at the center; to determine the first position of the first part of the key anchoring structure in the second image and the second position of the second part of the key anchoring structure; to determine the center of the line connecting the first position and the second position as the center of the second reference architecture spatial coordinate system, and to construct a three-dimensional direction vector of the second reference architecture spatial coordinate system based on the tangent plane of the line at the center.

[0144] Optionally, the third acquisition unit further includes a comparison unit, used to compare the first image region and the second image region in the object classification model to obtain the repair parameters of the target object, wherein the repair parameters are used to indicate the degree of repair of the target object; and to determine the repair classification result of the target object based on the repair parameters of the target object, wherein the repair classification result includes complete repair and incomplete repair.

[0145] Optionally, the comparison unit is further used to obtain a third sample set, wherein the third sample set includes a first sample subset and a second sample subset. The first sample subset includes multimodal sample images containing abnormal objects within the subject, and the second sample subset includes multimodal sample images collected after repairing the abnormal objects in the first sample subset. The initialized classification model is trained using the third sample set until an object classification model that satisfies the third convergence condition is obtained.

[0146] Optionally, the object recognition device further includes a fourth acquisition unit, used to acquire a first sample set and a second sample set; train the initialized segmentation model using the first sample set until a first segmentation model that satisfies the first convergence condition is obtained; and train the initialized segmentation model using the second sample set until a second segmentation model that satisfies the second convergence condition is obtained.

[0147] According to one aspect of this application, a computer-readable storage medium is provided, wherein a processor of a computer device reads computer instructions from the computer-readable storage medium, and executes the computer instructions to cause the computer device to perform the methods provided in the various alternative implementations described above.

[0148] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0149] S1, acquire the first image and the second image obtained by image acquisition of the subject to be identified, wherein the subject includes multiple objects with functional relationships, and the second image is an image acquired after the first image;

[0150] S2, input the first image into the first segmentation model and the second segmentation model, and input the second image into the second segmentation model. The first segmentation model is obtained by training the initialized segmentation model using the first sample set to segment the target object that meets the object anomaly condition from the image. The second segmentation model is obtained by training the initialized segmentation model using the second sample set to segment the structural framework that supports the object from the image. The first sample set includes first samples carrying the location labels of the target object's location, and the second sample set includes second samples carrying the location labels of each anchor structure in the structural framework.

[0151] S3, obtain the first location region information of the target object output by the first segmentation model, and obtain the anchoring information of the first architecture spatial coordinate system that matches the first image and the second architecture spatial coordinate system that matches the second image, determined based on the segmentation result output by the second segmentation model.

[0152] S4. Based on the first location region information of the target object, the anchoring information of the first architecture spatial coordinate system and the anchoring information of the second architecture spatial coordinate system, the repair classification result of the target object is identified.

[0153] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0154] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0155] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0156] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0160] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An object recognition method, characterized in that, include: A first image and a second image are obtained by acquiring images of the subject to be identified, wherein the subject includes multiple objects with functional relationships, and the second image is an image acquired after the first image; The first image is input into the first segmentation model and the second segmentation model, and the second image is input into the second segmentation model. The first segmentation model is obtained by training an initialized segmentation model using a first sample set to segment target objects that meet the object anomaly conditions from the image. The second segmentation model is obtained by training an initialized segmentation model using a second sample set to segment structural frames that support the object from the image. The first sample set includes first samples carrying location labels of the target object's location, and the second sample set includes second samples carrying location labels of each anchoring structure in the structural frame. Obtain the first location region information of the target object output by the first segmentation model, and obtain the anchoring information of the first architecture space coordinate system that matches the first image and the second architecture space coordinate system that matches the second image, based on the segmentation result output by the second segmentation model. Based on the first location region information of the target object's location, the anchoring information of the first architecture space coordinate system, and the anchoring information of the second architecture space coordinate system, the repair classification result of the target object is identified.

2. The method according to claim 1, characterized in that, The repair classification result of the target object, based on the first location region information of the target object's location, the anchoring information of the first architecture space coordinate system, and the anchoring information of the second architecture space coordinate system, includes: Obtain a first coordinate transformation matrix between the first architecture space coordinate system and the reference architecture space coordinate system, and a second coordinate transformation matrix between the reference architecture space coordinate system and the second architecture space coordinate system, wherein the reference architecture space coordinate system is determined based on the key anchoring structures in the structural framework of the main body; The first location region information of the target object is transformed according to the first coordinate transformation matrix to obtain the reference location region coordinates of the target object in the reference architecture space coordinate system; The coordinates of the reference position region are transformed according to the second coordinate transformation matrix to obtain the second position region information of the projection position of the target object in the second image; The first image region indicated by the first location region information in the first image and the second image region indicated by the second location region information in the second image are input into the object classification model, wherein the object classification model is used to determine the repair classification result of the target object based on the comparison result between the first image region and the second image region.

3. The method according to claim 2, characterized in that, After inputting the first image into the first segmentation model and the second segmentation model, and inputting the second image into the second segmentation model, the method further includes: The anchoring information of the first architecture spatial coordinate system is determined by the location of each anchoring structure in the first image output by the second segmentation model. Based on the first anchoring position of the key anchoring structure in the segmentation result of the first image, a first reference architecture spatial coordinate system is constructed. Calculate the first coordinate transformation matrix from the first architecture space coordinate system to the first reference architecture space coordinate system; The anchoring information of the second architecture spatial coordinate system is determined by the location of each anchoring structure in the second image output by the second segmentation model. Based on the second anchoring position of the key anchoring structure in the segmentation result of the second image, a second reference architecture spatial coordinate system is constructed; Calculate the second coordinate transformation matrix from the second reference architecture space coordinate system to the first architecture space coordinate system; The first reference architecture space coordinate system and the second reference architecture space coordinate system are the same coordinate system, and both are the reference architecture space coordinate system.

4. The method according to claim 3, characterized in that, The construction of a first reference architecture spatial coordinate system based on the first anchoring position of the key anchoring structure in the segmentation result of the first image includes: determining the first position of the first part of the key anchoring structure in the first image and the second position of the second part of the key anchoring structure; determining the center of the line connecting the first position and the second position as the center of the first reference architecture spatial coordinate system, and constructing a three-dimensional direction vector of the first reference architecture spatial coordinate system based on the tangent of the connecting line at the center of the circle; The construction of a second reference architecture spatial coordinate system based on the second anchoring position of the key anchoring structure in the segmentation result of the second image includes: determining the first position of the first part of the key anchoring structure in the second image and the second position of the second part of the key anchoring structure; determining the center of the line connecting the first position and the second position as the center of the second reference architecture spatial coordinate system, and constructing a three-dimensional direction vector of the second reference architecture spatial coordinate system based on the tangent of the connecting line at the center of the circle.

5. The method according to claim 2, characterized in that, The step of inputting the first image region indicated by the first location region information in the first image and the second image region indicated by the second location region information in the second image into the object classification model includes: In the object classification model, the first image region and the second image region are compared to obtain the repair parameters of the target object, wherein the repair parameters are used to indicate the degree of repair of the target object; The repair classification result of the target object is determined based on the repair parameters of the target object, wherein the repair classification result includes complete repair and incomplete repair.

6. The method according to claim 5, characterized in that, Before acquiring the first and second images obtained from image acquisition of the subject to be identified, the method further includes: Obtain a third sample set, wherein the third sample set includes a first sample subset and a second sample subset, the first sample subset includes multimodal sample images containing abnormal objects within the subject, and the second sample subset includes multimodal sample images collected after repairing the abnormal objects in the first sample subset; The initial classification model is trained using the third sample set until the object classification model that satisfies the third convergence condition is obtained.

7. The method according to any one of claims 1 to 6, characterized in that, Before inputting the first image into the first segmentation model and the second segmentation model, and before inputting the second image into the second segmentation model, the method further includes: Obtain the first sample set and the second sample set; The initial segmentation model is trained using the first sample set until the first segmentation model that satisfies the first convergence condition is obtained. The initial segmentation model is trained using the second sample set until the second segmentation model that satisfies the second convergence condition is obtained.

8. An object recognition device, characterized in that, include: The first acquisition unit acquires a first image and a second image obtained by image acquisition of the subject to be identified, wherein the subject includes multiple objects with functional relationships, and the second image is an image acquired after the first image; The segmentation unit inputs the first image into a first segmentation model and a second segmentation model, and inputs the second image into the second segmentation model. The first segmentation model is obtained by training an initialized segmentation model using a first sample set to segment target objects that meet the object anomaly conditions from the image. The second segmentation model is obtained by training an initialized segmentation model using a second sample set to segment structural frames that support the object from the image. The first sample set includes first samples carrying location labels of the target object's location, and the second sample set includes second samples carrying location labels of each anchoring structure in the structural frame. The second acquisition unit acquires the first location region information of the target object output by the first segmentation model, and acquires the anchoring information of the first architecture space coordinate system that matches the first image and the second architecture space coordinate system that matches the second image, determined based on the segmentation result output by the second segmentation model. The identification unit identifies the repair classification result of the target object based on the first location region information of the target object's location, the anchoring information of the first architecture space coordinate system, and the anchoring information of the second architecture space coordinate system.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images