Key point detection method and equipment

Abstract

Image-based keypoint detection method using convolutional neural networks (CNNs). When keypoints are obscured in an image, images obtained from additional imaging modalities such as depth and / or thermal images can be used in conjunction with the RGB image to reduce or minimize the effects of the obscuration. Images from other modalities can be used to determine conditioning parameters that are then applied to the weights of the CNN so that convolution operations can be performed in a modality-aware manner to improve the robustness, accuracy, and efficiency of keypoint detection.

Description

Technical Field

[0001] This application relates to image recognition technology. Background Art

[0002] Knowledge about the location of a patient's key anatomical structures plays a very important role in clinical settings. Using joint position as an example, information about the location of the joints can be used to build a model for determining the patient's body shape and / or posture, from which insights into the patient's positioning, physiological characteristics, appropriate radiation dose levels, etc. can be obtained. Joint positions can be automatically determined based on images of the patient captured during the medical procedure using deep learning (DL)-based object detection techniques. Typically, these techniques work by distinguishing the features of the image associated with the joint position (e.g., intensity, contrast, curvature, etc.) from the rest of the patient's body or the image background. As can be seen, these techniques rely on making a significant portion of the patient's body visible (e.g., uncovered) in the image. However, in many medical environments, the patient's body may be at least partially obscured or covered by medical equipment, radiation shielding, hospital gowns, etc., making it difficult to obtain an accurate estimate of the joint position using the above-mentioned techniques. Therefore, it is highly desirable to develop systems, methods, and devices with the ability to detect key points (e.g., joint positions) from images of the patient (even if these key points are partially or completely obscured in the image). Summary of the Invention

[0003] Described herein are systems, methods, and apparatus associated with detecting key points (e.g., joint locations) of an object based on an image of the object. The detection can be performed by obtaining a first image of the object including a first imaging characteristic, obtaining a second image of the object including a second imaging characteristic, and processing the first image using a convolutional neural network (CNN) based on at least the first imaging characteristic and the second imaging characteristic. The CNN can include a first convolutional layer configured to perform a first set of convolution operations on the first image based on the first imaging characteristic. The first convolutional layer can include a plurality of kernels, each of which can be associated with a corresponding set of weights, and performing the first set of convolution operations can include adjusting a set of weights associated with each of the plurality of kernels of the first convolutional layer based on the second imaging characteristic included in the second image.

[0004] In an example, the first image may include a red / green / blue (RGB) image of the object, and the second image may include a depth or thermal image of the object. In an example, one or more key points may be masked in the first image, and detection may be facilitated by the second image. In an example, the CNN may further include a second convolutional layer having a plurality of kernels, each kernel being associated with a corresponding second set of weights, and detection may be achieved by performing a second set of convolution operations on an output produced by the first convolutional layer, wherein the second set of weights associated with each of the plurality of kernels of the second convolutional layer is adjusted based on a second imaging characteristic. In an example, when the second set of convolution operations is performed on the output produced by the first convolutional layer, the second set of weights associated with each of the plurality of kernels of the second convolutional layer may not be adjusted based on the second imaging characteristic.

[0005] In an example, adjusting a set of weights associated with each of a plurality of kernels of a first convolutional layer may be performed as follows. For each pixel of a first image, a corresponding pixel in a second image is identified, an adjustment amount is determined based on a second imaging characteristic associated with the corresponding pixel in the second image, and a weight of the first convolutional layer to be applied to the pixel of the first image is adjusted according to the determined adjustment amount. In an example, each pixel of the first image may belong to a pixel block to be sampled by the kernel of the first convolutional layer, and the adjustment may be determined by determining a pixel block in the second image that corresponds to the pixel block to be sampled in the first image, and determining the adjustment based on the second imaging characteristic associated with two or more pixels of the corresponding pixel block in the second image.

[0006] In an example, the adjustments described herein may be indicated by a weight map or another suitable data structure. In an example, one or more key points of an object may be determined using a separate branch of a CNN, the separate branch of the CNN being configured to process the first image in conjunction with the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A more detailed understanding of the examples disclosed herein can be obtained from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0008] Figure 1 is a block diagram illustrating an example environment in which one or more embodiments described herein may be implemented.

[0009] Figure 2 is a block diagram illustrating an example of determining key anatomical points of a patient using a deep learning-based approach and multiple imaging modalities according to one or more embodiments described herein.

[0010] Figure 3A and Figure 3Bare block diagrams illustrating example convolution operations that may be performed by a conventional convolutional neural network and a pattern-aware convolutional neural network, respectively, according to one or more embodiments described herein.

[0011] Figure 4 is a flowchart illustrating an example process for training a neural network to perform a keypoint detection task according to one or more embodiments described herein.

[0012] Figure 5 is a block diagram illustrating example components of a device that may be used to perform keypoint detection tasks according to one or more embodiments described herein. DETAILED DESCRIPTION

[0013] The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0014] Figure 1 FIG1 is a diagram illustrating an example environment 100 associated with one or more embodiments described herein. Environment 100 may be part of a medical facility, such as a scanning room (e.g., magnetic resonance imaging (MRI), X-ray, computed tomography (CT), etc.), an operating room (OR), a rehabilitation facility, a fitness center, etc. Environment 100 may be equipped with one or more sensing devices 102, such as one or more digital cameras, configured to capture images of a patient 104 within environment 100. Sensing devices 102 may be installed at various locations within environment 100 and may be communicatively coupled to processing device 106 and / or other devices of environment 100 via a communication network 108. Each sensing device 102 may include one or more sensors, such as a 2D vision sensor (e.g., a 2D camera), a 3D vision sensor (e.g., a 3D camera), a red, green, and blue (RGB) sensor, a depth sensor, an RGB plus depth (RGB-D) sensor, a thermal sensor (e.g., an infrared (FIR) or near infrared (NIR) sensor), a radar sensor, and / or other types of image capture devices or circuits. Thus, according to one or more embodiments described herein, depending on the type of sensors included therein, the sensing device 102 may be able to generate images of the patient 104 having different imaging characteristics (such as RGB (e.g., first imaging characteristic) images, depth (e.g., second imaging characteristic) images, thermal (e.g., third imaging characteristic) images, etc.), which may be used to determine key anatomical features of the patient (e.g., joint position).

[0015] The processing unit 106 can be configured to process the images captured by the sensing device and determine the locations of key anatomical structures of the patient 104 based on the images. These anatomical locations may include, for example, joint locations (e.g., elbow, shoulder, wrist, knee, ankle, etc.), which can be used to construct a model (e.g., a 2D or 3D human body mesh model) for determining the patient's body shape and / or posture during a medical procedure (e.g., an MRI, X-ray, or CT procedure). The processing unit 106 can be configured to perform key point detection and / or model building tasks using a deep learning-based approach. For example, the processing unit 106 may include one or more processors (e.g., a graphics processing unit (GPU)) and a memory configured to implement an artificial neural network (ANN) pre-trained to perform the above-mentioned tasks. The processing unit 106 can be configured to store the processing results (and / or the images captured by the sensing device 102) in a database 110 that is communicatively coupled to the processing unit 106 (e.g., via a communication link 108).

[0016] Instead of or in addition to transmitting images to the processing unit 106 and causing the processing unit 106 to perform key point detection and / or model building operations, one or more sensing devices 102 may include a functional unit (e.g., a processor) configured to perform some or all of the operations. One or more sensing devices 102 may be communicatively coupled to other devices located inside or outside the environment 100, for example, via corresponding communication circuits and communication links 108 included in the sensing devices 102, so that the other devices can facilitate the performance of the tasks described herein. The sensing devices 102 may also be communicatively coupled to a database 110 via the communication link 108 and configured to store images and / or processing results generated by the sensing devices to the database 110. Each sensing device 102 may be a standalone device, or may be part of an edge device configured to perform tasks such as remote patient monitoring, automated care delivery, distributed information collection and analysis, and the like.

[0017] When inside the environment 100, the patient 104 may be covered (e.g., occluded or otherwise obscured) by another object, such as the cover sheet 112. Consequently, portions of the patient's body (e.g., knees and / or ankles) covered by the sheet 112 may not be visible in images (e.g., RGB images) of the patient 104 captured by one or more sensing devices 102 (e.g., RGB image sensors). The processing unit 106 and / or the sensing device 102 (e.g., corresponding functional units of the sensing device) may be configured to detect key anatomical features (e.g., joint positions) of the patient 104 based on these images, even if the anatomical features are occluded or obscured (or otherwise not visible) by another object in the image. For example, the processing unit 106 and / or the sensing device 102 (e.g., corresponding functional units of the sensing device) may be configured to compensate for the occlusion or obscuration by utilizing information collected from multiple image modalities. In one or more embodiments, this may include detecting key points of the patient based on one or more of depth, thermal (e.g., infrared), or radar images in combination with the RGB images. In one or more embodiments, this may include performing a weighted mode-aware convolution operation on an input image produced by a first imaging mode (e.g., an image configured to produce a first imaging characteristic) based on information (e.g., depth, thermal, etc.) obtained from a second imaging mode (e.g., an image configured to produce a second imaging characteristic). In one or more embodiments, this may include calculating a plurality of adjustment parameters (e.g., in the form of a weight map) based on the second image, and adjusting parameters of a neural network used to process the first image (e.g., weights associated with one or more convolution kernels) based on the adjustment parameters.

[0018] Figure 2 An example of using a deep learning-based approach and multiple imaging modalities to determine key anatomical features of a patient is illustrated. The figure will be described using joint locations as an example, but it should be understood that the techniques described herein can also be applied to other anatomical structures of the human body. Moreover, even though a single artificial neural network (ANN) is shown in the figure, it should be understood that the techniques described herein can also be implemented using multiple ANNs.

[0019] like Figure 2 As shown, the ANN 202 can be configured to obtain (e.g., receive) a first image 204 of a patient generated by a first imaging modality, determine one or more joint positions of the patient based at least on the first image 204, and generate an output 206 indicating the one or more determined joint positions. In an example, the first image 204 can be an RGB image, and the first imaging modality can include an RGB sensing device, such as Figure 12. The sensing device 102 is shown. Thus, the first image 204 can be characterized by one or more RGB characteristics associated with the patient (e.g., one or more first imaging characteristics). In an example, one or more portions of the patient's body (e.g., joints) may be obscured by another object (e.g., a cover sheet) in the image 204. Consequently, image information associated with the obscured portion of the patient's body may be unavailable to the ANN 202 or may be mixed with image information of the obscuring object, thereby potentially affecting the accuracy of the joint position prediction performed by the ANN 202.

[0020] The ANN 202 can also be configured to obtain information associated with a second image 208 of the patient generated by the second imaging modality and utilize the information to improve the accuracy and / or efficiency (e.g., speed) of the joint position detection operation. For example, the second image 208 can be a depth image, a thermal image (e.g., including an infrared image), a radar image, and / or a combination thereof, and the second imaging modality can include a depth, thermal, or radar sensing device, such as Figure 1 2. The sensing device 102 is shown. It can be seen that the second image 208 can be characterized by one or more depth, thermal, or radar-sensitive characteristics associated with the patient (e.g., one or more second imaging characteristics). The ANN 202 can be configured to obtain (e.g., calculate or receive) a plurality of adjustment parameters reflecting the depth, thermal, and / or radar characteristics captured in the second image 208 and apply these adjustment parameters to the processing of the first image 204.

[0021] In an example, the ANN 202 may include a plurality of convolutional layers, one or more pooling layers, and / or one or more fully connected layers. In an example, each convolutional layer may include a plurality of convolution kernels or filters that are configured to identify specific features in the input image 204 that may represent joint positions, for example, based on a first imaging characteristic associated with the input image 204. The convolution kernels or filters may be associated with corresponding weights W that, when applied to the input, may produce an output indicating whether a certain visual feature or pattern has been detected. Using a two-dimensional (2D) convolution as an example, the convolution operation may be represented by the following equation:

[0022]

[0023] Where R can represent a grid corresponding to a convolution kernel (e.g., a 3×3 convolution kernel), x(p0+p n ) can represent the pixel (p0+p n ), y(p0) can represent the output value at pixel p0, and w(p n ) can represent the offset p in the convolution kernel n The weight of .

[0024] The convolution operation performed by the ANN 202 may be followed by batch normalization and / or linear or nonlinear activation, and the features extracted by the convolution layer may be downsampled by one or more pooling layers (e.g., using a 2×2 window and a stride of 2) to reduce the redundancy and / or size of the features (e.g., by a factor of 2). As a result of the convolution and / or downsampling operations, a corresponding feature representation of the input image 204 may be obtained, for example, in the form of one or more feature maps or feature vectors.

[0025] ANN 202 may also include multiple transposed convolution layers and / or one or more non-pooling layers. Through these layers, ANN 202 may perform a series of upsampling and / or transposed convolution operations based on the feature maps or feature vectors generated by the above-mentioned downsampling operations. For example, ANN 202 may upsample the feature representation based on the pooling index stored in the downsampling stage (e.g., using a 3×3 transposed convolution kernel with a stride of 2) to restore the features extracted from the input image 204 in the output 206.

[0026] When a joint of a patient is obscured in image 204, it may become challenging to accurately determine the location of the joint because image information (e.g., color, intensity, etc.) that can be used to distinguish the joint position from other parts of the image is missing from the image. ANN 202 can be configured to compensate for the obstruction or occlusion using additional information provided by image 208 (e.g., a second imaging characteristic). For example, ANN 202 can be configured to modify the convolution operation described herein based on adjustment parameters determined from image 208 (e.g., from the second imaging characteristic of image 208). These adjustment parameters can be determined as, for example, a pattern weight map 210, and can be operated to distinguish the obscured or covered joint position from other parts of the patient's body or other objects (e.g., a piece covering the joint) in image 204. Using a depth image as an example of image 208, such an image may include information indicating the depth characteristics of the joint (e.g., pixels of the joint may be associated with corresponding depth values). Thus, even if a joint is occluded by another object and becomes indistinguishable from the other object in image 204 (e.g., an RGB image), the joint can still be distinguished from the other object based on image 208 because pixels associated with the joint may have different depth values ​​than pixels of the other object (e.g., the joint may be below or behind the other object). This is also true for thermal imaging modes or other imaging modes, as long as these imaging modes can provide additional information to distinguish key points of interest.

[0027] In an example, the ANN 202 can be configured to perform a pattern-aware convolution operation as follows. Denoting the weight map (e.g., or another data structure including the adjustment parameters described herein) as MD, the convolution operation shown above can be modified based on the following equation:

[0028]

[0029] The weight map (eg, adjustment parameters) may be derived based on the image 208 using various techniques. For example, the weight map may be derived based on the pixel position (p) in the image 204. i , p j ) The associated adjustment parameters can be determined as follows:

[0030]

[0031] Wherein, C can be a predetermined constant value, and I(p i ) and I(p j ) can represent the corresponding pixel position p in image 208 i and p j The corresponding characteristic value (e.g., depth, heat, intensity, etc.) associated with the image 204 may be a pixel block to be sampled by the convolution kernel, as indicated by (1), and the adjustment parameter may be determined by identifying a pixel block in the second image that corresponds to the pixel block in the first image, and determining the adjustment parameter based on a second imaging characteristic associated with pixels (e.g., two or more pixels) of the corresponding pixel block in the second image, as indicated by (2) and (3).

[0032] The ANN 202 can be configured to modify all or a subset of the convolution operations using the pattern-aware convolution operations described herein. For example, the ANN 202 can be configured to modify the convolution operations of each convolution layer or only a subset of the convolution layers. Further, as the image 204 is processed by one or more layers, adjustment parameters for the weights to be applied to one or more convolution layers can be determined in real time (e.g., dynamically).

[0033] Using the techniques described herein, the number of parameters associated with ANN 202 can be kept approximately the same as compared to conventional convolutional networks, thereby preventing ANN 202 from becoming too complex. Further, the techniques may not require the addition of additional branches to ANN 202, which may result in lower complexity and / or faster processing speeds compared to multi-branch networks. Thus, the operations described herein may be implemented on devices with full computational capabilities (e.g., Figure 1 106), or on a sensing or edge device with reduced computing power (e.g., Figure 1 The method is performed on the sensing device 102 shown.

[0034] Figure 3AAn example convolution operation that can be performed by a conventional convolutional neural network is illustrated. As shown in the figure, processing an input X obtained from a first imaging mode (for example, which may have a first imaging characteristic) by a convolutional layer may include: assigning corresponding weights (W1, W2...W n ), bias value b and / or activation function f are applied to the input to obtain the output Y. Figure 3B An example convolution operation that can be performed by a mode-aware convolutional neural network as described herein is illustrated. As shown, processing an input X obtained from a first imaging mode by a convolutional layer may include adjusting corresponding weights (W1, W2, ..., W2) associated with the convolutional layer using adjustment parameters determined based on information obtained from a second imaging mode (e.g., corresponding to a second imaging characteristic). n ), and applies the adjusted weights, bias values ​​b and / or activation function f to the input to obtain the output Y.

[0035] Figure 4 Illustrate a method for training a neural network (e.g., Figure 2 202) to perform keypoint detection operations described herein. Training can be performed using images collected from multiple imaging modalities including, for example, RGB, thermal, depth, radar, etc. Process 400 can begin at 402, and at 404, initial parameters of the neural network (e.g., weights associated with various filters or kernels of the neural network) can be initialized. The parameters can be initialized, for example, based on samples collected from one or more probability distributions or parameter values ​​of another neural network with a similar architecture. At 406, the neural network can receive a first image associated with an object (e.g., a patient) (e.g., an image having a first imaging characteristic such as an RGB characteristic) and a second image associated with the object (e.g., an image having a second imaging characteristic such as a depth characteristic), and process the first image through a first convolutional layer. The processing can include adjusting parameters (e.g., weights) associated with the first convolutional layer based on adjustment parameters (e.g., a weight map), the adjustment parameters being calculated based on the second image. At 408, the neural network may process the output generated by the first layer through one or more additional convolutional layers, and at each convolutional layer, the neural network may adjust the parameters (e.g., weights) associated with the layer based on the adjustment parameters calculated based on the corresponding pixels of the second image. At 410, the results predicted by the neural network may be compared with a gold standard to determine the adjustments that need to be made to the neural network parameters. The adjustments may be determined, for example, based on a loss function (e.g., MSE, L1, L2, etc.) and a gradient descent (e.g., stochastic gradient descent) associated with the loss function.

[0036] At 412, the neural network may apply adjustments to the currently assigned network parameters, for example, through a backpropagation process. At 414, the neural network may determine whether one or more training termination criteria have been met. For example, if the neural network has completed a predetermined number of training iterations, if the difference between the predicted result and the gold standard value is below a predetermined threshold, or if the change in the value of the loss function between two training iterations is below a predetermined threshold, the neural network may determine that the training termination criteria have been met. If, at 414, it is determined that the training termination criteria have not been met, the neural network may return to 406. If, at 414, it is determined that the training termination criteria have been met, the neural network may end the training process 400 at 416.

[0037] For simplicity of illustration, the training steps are depicted and described herein in a particular order. However, it should be understood that the training operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that not all operations that may be included in the training process are depicted and described herein, and not all illustrated operations need to be performed.

[0038] The systems, methods, and / or devices described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 5 is a block diagram illustrating an example device 500 that can be configured to perform key point detection operations described herein. As described herein, the device 500 can be a standalone device (such as Figure 1 The processing unit 106 shown in FIG. 1 ), or the device 500 may be an edge device configured to perform tasks such as remote patient monitoring, automated care delivery, distributed information collection and analysis, etc. The device 500 may also be part of a sensing device (such as the sensing device 102 ) that may also include an image sensor.

[0039] like Figure 5 As shown, the device 500 may include a processor (e.g., one or more processors) 502, which may be a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, a reduced instruction set computer (RISC) processor, an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or any other circuit or processor capable of performing the functions described herein. The device 500 may also include communication circuitry 504, a memory 506, a mass storage device 508, an input device 510, and / or a communication link 512 (e.g., a communication bus), through which one or more components shown in the figure can exchange information.

[0040] The communication circuit 504 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including a local area network (LAN), a wide area network (WAN), the Internet, a wireless data network (e.g., Wi-Fi, 3G, 4G / LTE, or 5G network). The memory 506 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions that, when executed, cause the processor 502 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc. The mass storage device 508 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROM or DVD-ROM disks, etc., on which instructions and / or data may be stored to facilitate the operation of the processor 502. The input device 510 may include a keyboard, a mouse, a voice control input device, a touch-sensitive input device (eg, a touch screen), etc., for receiving user input of the device 500 .

[0041] It should be noted that the apparatus 500 can operate as a standalone device or can be connected (e.g., networked or clustered) with other computing devices to perform the functions described herein. Figure 5 Only one example of each component is shown in the figure, and those skilled in the art will also understand that the device 500 may include multiple examples of one or more components shown in the figure.

[0042] Although the present disclosure has been described in terms of certain embodiments and generally associated methods, variations and transformations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit the present disclosure. Other changes, substitutions, and variations are also possible without departing from the spirit and scope of the present disclosure. In addition, unless otherwise specifically stated, discussions utilizing terms such as "analyze," "determine," "enable," "identify," "modify," etc. refer to the actions and processes of a computer system or similar electronic computing device, which manipulate and transform data represented as physical (e.g., electronic) quantities within the registers and memories of a computer system into other data represented as physical quantities within the computer system memory or other such information storage, transmission, or display device.

[0043] It should be understood that the above description is intended to be illustrative, rather than restrictive. After reading and understanding the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the present disclosure should be determined with reference to the full scope of equivalents to which the appended claims and such claims are given.

Claims

1. A method for detecting key points of an object, characterized in that The method comprises: obtaining a first image of the object, wherein the first image includes a first imaging characteristic of the object; obtaining a second image of the object, wherein the second image includes a second imaging characteristic of the object; and The first image is processed by a convolutional neural network (CNN) based on at least the first imaging characteristic and the second imaging characteristic to determine one or more key points of the object, wherein The convolutional neural network includes a first convolutional layer configured to perform a first set of convolution operations on the first image based on the first imaging characteristic of the object; The first convolutional layer includes a plurality of kernels, each kernel being associated with a corresponding set of weights; and Performing the first set of convolution operations on the first image includes: adjusting the set of weights associated with each of the plurality of kernels of the first convolutional layer based on the second imaging characteristic of the object; Adjusting the set of weights associated with each of the plurality of kernels of the first convolutional layer based on the second imaging characteristic of the object includes: For each pixel p of the first image: identifying corresponding pixels in the second image; determining an adjustment amount based on the second imaging characteristic associated with the corresponding pixel in the second image; and The weight of the first convolutional layer applied to the pixel p is adjusted according to the adjustment amount.

2. The method according to claim 1, characterized in that The first image includes a red / green / blue (RGB) image of the object, and the second image includes a depth or thermal image of the object.

3. The method according to claim 1, wherein The one or more key points are masked in the first image, or the one or more key points include joint positions.

4. The method according to claim 1, wherein The convolutional neural network also includes a second convolutional layer, which includes a plurality of kernels, each kernel being associated with a corresponding second set of weights, and the method further includes: performing a second set of convolution operations on the output of the first convolutional layer, and wherein, during the performance of the second set of convolution operations, the second set of weights associated with each of the plurality of kernels of the second convolutional layer is adjusted based on the second imaging characteristic of the object.

5. The method according to claim 1, wherein The convolutional neural network also includes a second convolutional layer, which includes a plurality of kernels, each kernel being associated with a corresponding second set of weights, and the method further includes: performing a second set of convolution operations on the output of the first convolutional layer, and wherein the second set of weights associated with each of the plurality of kernels of the second convolutional layer is not adjusted based on the second imaging characteristic of the object during the performance of the second set of convolution operations.

6. The method according to claim 1, characterized in that The convolutional neural network includes a plurality of additional convolutional layers each associated with a respective set of weights, the method further comprising: performing a convolution operation on the first image via the plurality of additional convolutional layers, and wherein the respective set of weights associated with each of the plurality of additional convolutional layers is adjusted based on the second imaging characteristic of the object during the performance of the convolution operation.

7. The method according to claim 1, characterized in that Respective pixels of the first image belong to a pixel block to be sampled using the kernel of the first convolutional layer, and wherein determining the adjustment amount based on the second imaging characteristic associated with the corresponding pixel in the second image comprises: determining a pixel block in the second image corresponding to the pixel block to be sampled in the first image; and The adjustment amount is determined based on the second imaging characteristic associated with two or more pixels of the corresponding pixel block in the second image.

8. The method according to claim 1, characterized in that The one or more keypoints of the object are determined using a separate branch of the convolutional neural network, wherein the separate branch of the convolutional neural network is configured to process the first image in combination with the second image.

9. A device configured to detect key points of an object, characterized in that include: One or more processors configured to: obtaining a first image of the object, wherein the first image includes a first imaging characteristic of the object; obtaining a second image of the object, wherein the second image includes a second imaging characteristic of the object; and The first image is processed by a convolutional neural network (CNN) based on at least the first imaging characteristic and the second imaging characteristic to determine one or more key points of the object, wherein The convolutional neural network includes a first convolutional layer configured to perform a first set of convolution operations on the first image based on the first imaging characteristic; The first convolutional layer includes a plurality of kernels, each kernel being associated with a corresponding set of weights; and The one or more processors being configured to perform the first set of convolution operations on the first image includes: the one or more processors being configured to adjust the set of weights associated with each of the plurality of kernels of the first convolutional layer based on the second imaging characteristic of the object; Adjusting the set of weights associated with each of the plurality of kernels of the first convolutional layer based on the second imaging characteristic of the object includes: For each pixel p of the first image: identifying corresponding pixels in the second image; determining an adjustment amount based on the second imaging characteristic associated with the corresponding pixel in the second image; and The weight of the first convolutional layer applied to the pixel p is adjusted according to the adjustment amount.

Images